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Executive  Summary 


Introduction.  The  Electronic  Commerce  Resource  Centers  (ECRC)  Program  is  a  DoD 
funded  network  of  centers  established  to  help  small-  and  medium-sized  enterprises 
(vendors)  and  Government  organizations  implement  and  use  electronic  commerce 
technologies.  The  ECRC  program  focuses  on  vendors  who  are  conducting  or  want  to 
conduct  business  with  DoD.  The  ECRC  program  is  composed  of  11  regional  centers, 
2  team  integrators,  and  1  ECRC  technology  hub.  The  regional  ECRCs  help  vendors 
and  Government  organizations  learn  about  and  adopt  electronic  commerce  technologies 
through  outreach,  education  and  training,  consultation,  and  technical  support.  The 
team  integrators  communicate,  coordinate,  plan,  and  oversee  the  operations  of  the 
regional  ECRCs.  The  technology  hub  coordinates,  develops,  and  validates  needed 
electronic  commerce  technology  and  makes  it  available  to  the  regional  ECRCs. 
Congress  appropriated  $108.6  million  for  ECRC  program  activities  from  FYs  1993 
through  1995,  and  DoD  budgeted  $83.7  million  for  the  ECRC  program 
FYs  1996  through  1998,  $15  million  of  which  is  to  establish  5  new  ECRCs.  First  the 
Air  Force  then  the  Defense  Advanced  Research  Projects  Agency  were  responsible  for 
management  and  oversight  of  the  ECRC  program  until  October  1,  1996.  In  FY  1997, 
the  ECRC  program  was  transferred,  by  direction  of  Congress,  to  the  Deputy  Under 
Secretary  of  Defense  (Logistics)  and  management  and  oversight  was  delegated  to  the 
Defense  Logistics  Agency. 

Audit  Objectives.  The  primary  audit  objective  was  to  review  the  management  of  the 
ECRCs.  Specifically,  we  sought  to  determine  whether  the  ECRC  program  provided 
efficient  and  cost-effective  training  and  outreach  to  Government  vendors  on  the 
implementation  and  use  of  electronic  commerce/electronic  data  interchange  (EC/EDI) 
technologies.  Another  audit  objective  was  to  evaluate  the  adequacy  of  the  management 
control  program  as  applicable  to  the  ECRC  program.  However,  we  did  not  review  the 
management  control  program  because  ongoing  Inspector  General,  DoD, 
Project  No.  5AB-(X)52,  "Audit  of  the  Management  and  Administration  of  Research 
Projects  Funded  by  the  Defense  Advanced  Research  Projects  Agency,"  will  discuss  the 
adequacy  of  the  Defense  Advanced  Research  Projects  Agency  management  control 
program. 

Audit  Results.  The  ECRC  program  has  not  been  efficient  or  cost-effective  in 
promoting  the  implementation  or  increased  use  of  EC/EDI  technologies  between 
Government  organizations  and  vendors.  Ninety-one  percent  of  the  customers  that 
received  ECRC  support  did  not  implement  or  increase  their  use  of  EC/EDI 
technologies  to  conduct  business  with  the  Federal  Government.  As  a  result,  DoD 
obligated  approximately  $81.5  million  for  FY  1994  through  the  first  quarter  of 
FY  1996  that  did  not  greatly  increase  the  implementation  and  use  of  K/EDI 
technologies.  The  $81.5  million  obligation  resulted  in  outreach,  training,  or 
consultation  with  12,764  ECRC  customer  organizations  but  about  1,102  increased  their 
use  of  EC/EDI  technologies.  Averaging  that  cost  over  the  1,102  organizations,  DoD 
spent  about  $74,(X)0  for  each  organization  that  implemented  or  increased  its  use  of 
EC/EDI  to  do  business  with  the  Government.  Averaging  the  $81.5  million  over  the 


entire  12,764  organizations,  DoD  spent  about  $6,400  on  each  organization  even  though 
91  percent  of  them  may  not  implement  or  increase  their  use  of  EC/EDI  in  the  future. 
In  addition,  about  34  percent  of  FYs  1995  and  1996  ECRC  budgets  were  consumed  by 
administrative  and  non-core  function  activities.  See  Part  I  for  a  discussion  of  the  audit 
results. 

Summary  of  Recommendations.  We  recommend  that  the  Deputy  Under  Secretary  of 
Defense  (Logistics)  seek  additional  time  to  implement  the  congressional  direction  to 
establish  five  new  ECRC  sites.  We  also  recommend  that  the  Deputy  Under  Secretary 
of  Defense  (Logistics)  streamline  the  multi-layered  ECRC  management  structure; 
re-evaluate  contracts  used  to  acquire  ECRC  support  and  justify  related  contract  fees; 
redirect  the  ECRC  program  with  focus  on  getting  DoD  procurement  offices  and 
vendors  to  use  EC/EDI  technologies;  establish  contractor  performance  measures; 
establish  a  Government-wide  EC/EDI  Integrated  Process  Team  to  optimize 
collaborative  efforts;  coordinate  ECRC  efforts  with  Defense  Logistics  Agency  managed 
Procurement  Technical  Assistance  Center  efforts;  and  seek  authorization  to  eliminate 
the  congressionally-directed  ECRC  technology  hub. 

Management  Comments  and  Audit  Response.  The  Deputy  Under  Secretary  of 
Defense  (Logistics)  did  not  respond  to  a  draft  of  this  report.  Therefore,  we  request  the 
Deputy  Under  Secretary  of  Defense  (Logistics)  to  provide  comments  by 
March  12,  1997. 
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Part  I  -  Audit  Results 


Audit  Results 


Audit  Background 


Electronic  Commerce/Electronic  Data  Interchange.  Electronic  commerce 
(EC)  is  the  paperless  exchange  of  business  information  using  technologies  such 
as  electronic  mail,  computer  bulletin  boards,  and  electronic  data  interchange 
(EDI).  EDI  is  integral  to  electronic  commerce  in  that  it  is  the  computer-to- 
computer  exchange  of  business  information  using  a  standardized  electronic 
format. 

The  Federal  Acquisition  Streamlining  Act  of  1994.  The  Federal  Acquisition 
Streamlining  Act  (the  Act)  simplified  and  streamlined  the  Federal  acquisition 
process  by  requiring  the  Government  to  transition  from  a  labor-intensive, 
paper-based  acquisition  process  to  a  more  efficient  process  using  EC/EDI 
technologies.  The  Act  ftirther  emphasized  the  need  for  small-  and  medium¬ 
sized  enterprises  (vendors)  to  migrate  toward  EC,  and  encouraged  the  use  of  the 
Federal  Acquisition  Computer  Network  (FACNET)  to  conduct  small  purchases 
electronically.  FACNET  is  an  EC/EDI  communications  network  being 
developed  to  enhance  access  to  DoD  procurement  information  for  vendors,  and 
to  provide  a  standard  format  for  the  exchange  of  acquisition  information 
between  Federal  Government  agencies  and  vendors.  The  Act  requires 
FACNET  to  be  implemented  Government-wide  no  later  than  January  2000  (the 
Act  originally  required  that  FACNET  be  implemented  by  January  1997). 

Electronic  Commerce  Resource  Centers  Program.  The  Electronic  Commerce 
Resource  Centers  (ECRC)  Program  (formerly  known  as  the  Continuous 
Acquisition  and  Life-cycle  Support  (CALS)  Shared  Resource  Center  Program) 
is  a  DoD-funded  network  of  centers  established  to  help  vendors  and 
Government  organizations  implement  and  use  EC/EDI  technologies. 
Specifically,  the  ECRCs  are  to  work  with  DoD  vendors;  vendors  to  large  DoD 
prime  contractors  or  their  supplier  network;  and  vendors  involved  in 
manufacturing  who  want  to  do  business  with  DoD,  other  Federal  Government 
organizations,  or  with  companies  that  do  business  with  the  Federal  Government. 

The  ECRC  program  consists  of  11  regional  ECRCs,  2  team  integrators,  and 
1  technology  hub.  The  mission  of  the  regional  ECRCs  is  to  help  vendors  and 
Government  organizations  learn  and  adopt  EC/EDI  technologies  through 
outreach,  education  and  training,  consultation,  and  technical  support  activities 
(referred  to  by  the  ECRCs  as  core  functions).  The  two  team  integrators. 
Concurrent  Technologies  Corporation  (CTC),  and  the  Cleveland  Advanced 
Manufacturing  Program  (CAMP),  coordinate,  plan,  and  oversee  operation  of 
the  regional  ECRCs  on  their  respective  teams.  CTC  is  responsible  for  six 
regional  ECRCs  and  CAMP  is  responsible  for  five  regional  ECRCs.  CTC  also 
operates  the  technology  hub  (the  hub)  for  the  ECRC  program.  The  hub  is 
supposed  to  coordinate,  develop,  and  validate  technologies  to  solve  problems 
preventing  the  implementation  and  use  of  EC/EDI .  The  hub  is  also  supposed  to 
provide  solutions  to  the  regional  ECRCs  who,  in  turn,  distribute  the  solutions  to 
vendors  and  Government  organizations. 

ECRC  Program  History.  In  FY  1991,  Congress  established  the  first  ECRC  at 
Johnstown,  Pennsylvania,  and  directed  the  Air  Force  to  serve  as  the  program 
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executive  agent.  Between  FYs  1992  and  1995,  Congress  established  and 
designated  the  specific  locations  of  10  additional  ECRCs.  In  FY  1991, 
Congress  designated  the  ECRC  program  a  "congressional  special  interest 
program,"  which  means  that  program  funds  cannot  be  reduced  without  prior 
approval  from  Congress.  Congress  also  required  DoD  to  award  sole-source 
contracts  to  three  contractors  to  operate  five  of  the  ECRCs.  Figure  1.  shows 
the  geographic  locations  of  the  11  ECRCs  and  denotes  the  regions  in  which  they 
operate. 


Figure  1.  Geographic  Locations  and  Regions  of  the  11  ECRCs 

In  FY  1994,  Congress  transferred  ECRC  management  and  oversight 
responsibilities  to  Defense  Advanced  Research  Projects  Agency  (DARPA). 
DARPA  renamed  the  Continuous  Acquisition  and  Life-cycle  Support  Shared 
Resource  Center  program  "the  ECRC  program"  to  further  emphasize  EC  and  to 
focus  the  program  on  helping  vendors  to  implement  and  use  EC/EDI 
technologies.  Congress  also  directed  DARPA  to  establish  the  hub.  In  1995,  in 
committee  reports  accompanying  the  FY  1996  DoD  Appropriations  Act, 
Congress  directed  that  DoD  transfer  ECRC  program  management  to  the 
Defense  Logistics  Agency  (DLA),  beginning  in  FY  1997.  Subsequently, 
Congress  reaffirmed  its  support  for  management  oversight  of  the  program  by 
the  Deputy  Under  Secretary  of  Defense  (Logistics),  in  the  Conference  Report 
accompanying  the  Onmibus  Consolidated  Appropriations  Act  for  FY  1997,  in 
order  to  fully  capitalize  on  and  expand  the  use  of  EC  technologies.  The  Deputy 
Under  Secretary  of  Defense  (Logistics)  delegated  ECRC  program  management 
responsibilities  to  DLA. 

ECRC  Program  Funding.  The  ECRC  program  is  funded  within  the  Research, 
Development,  Test,  and  Evaluation  Defense  appropriation.  The  ECRC  program 
is  forward  funded,  which  means  that  prior  year  funds  are  used  to  pay  current 
year  expenses.  For  example,  FY  1997  contracts  will  be  funded  by  amounts 
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budgeted  in  FY  1996.  As  a  result,  FY  1996  ECRC  funds  will  not  be  obligated 
until  the  FY  1997  ECRC  contracts  are  awarded.  Table  1  shows  ECRC  funding 
by  Fiscal  Year. 


Table  1.  ECRC  Funding  by  Fiscal  Year 
(Funds  in  MUIions) 

Fiscal  Budgeted  Authorized  Obligated 

Year  Amount  Amount  Amount 


1993 

$31.9 

$31.6 

1994 

— 

$43.0 

$42.8 

1995 

— 

$33.7 

$30.6 

1996 

$33.0 

— 

1997 

$35.7 

— 

— 

1998 

$15.0 

— 

— 

Total 

$83.7 

$108.6 

$105.0 

DoD  obligated  approximately  $105  million  of  the  $108.6  million  authorized  for 
ECRC  program  activities  from  FYs  1993  through  1995,  and  Congress  budgeted 
$83.7  million  for  the  ECRC  program  for  FYs  1996  through  1998.  Of  the 
$83.7  million  budgeted  for  FYs  1996  through  1998,  the  "Omnibus  Consolidated 
Appropriations  Act,  1997,"  authorized  $15  million  for  the  establishment  and 
operation  of  five  additional  regional  ECRCs  during  FY  1997. 


Audit  Objective 


The  overall  objective  was  to  evaluate  the  management  of  the  ECRCs.  The 
specific  objective  was  to  determine  whether  the  ECRC  program  provided 
efficient  and  cost-effective  training  and  outreach  to  Government  vendors  on  the 
implementation  of  EC/EDI  technologies.  See  Appendix  A  for  a  discussion  of 
the  audit  scope  and  methodology.  See  Appendix  B  for  a  summary  of  prior  audit 
coverage  related  to  the  audit  objectives. 
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ECRC  Program  Effectiveness 

To  date,  the  ECRC  program  has  not  been  efficient  or  cost-effective  in 
promoting  the  implementation  or  increased  use  of  EC/EDI  technologies 
between  Government  organizations  and  vendors.  DoD  spent  about 
$74,000  on  the  average  for  each  organization  that  implemented  or 
increased  its  use  of  EC/EDI.  The  ECRC  program  is  not  efficient  or 
cost-effective  because: 

o  About  8.6  percent  of  the  organizations  that  received  ECRC 
support  implemented  or  increased  their  use  of  EC/EDI  technologies  to 
conduct  business  with  the  Federal  Government. 

0  ECRC  core  functions  overlap  services  that  are  available 
elsewhere. 

0  The  technology  hub  did  not  develop  or  provide  any  unique 
solutions  to  vendors'  EC/EDI  implementation  problems. 

o  The  program  infrastructure  consumed  about  34  percent  of  the 
ECRC  budget  for  program  management  and  non-core  function  activities. 

o  Congress  shifted  organizational  control  of  the  program  and 
allowed  a  significant  level  of  contractor  autonomy  that  impeded  DoD's 
ability  to  manage  this  program. 

o  ECRC  contracts  did  not  provide  incentive  for  contractors  to 
use  efficient,  cost-effective  methods  and  practices. 

As  a  result,  DoD  obligated  approximately  $81.5  million  from  FY  1994 
through  the  first  quarter  of  FY  1996  but  did  not  significantly  increase  the 
implementation  and  use  of  EC/EDI  technologies.  In  addition.  Congress 
earmarked  $83.7  million  for  FYs  1996  through  1998  to  continue  the 
ECRC  program. 


ECRC  Program 


In  May  1988,  the  Deputy  Secretary  of  Defense  emphasized  the  importance  of 
EDI  in  a  memorandum,  "Electronic  Data  Interchange  of  Business-Related 
Transactions,"  which  stated  that  EDI  was  to  become  the  "way  of  doing 
business"  for  DoD  in  the  future.  In  1991,  Congress  established  the 
ECRC  program  to  help  DoD  realize  the  Deputy  Secretary's  vision.  According 
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to  the  ECRC  Program  Operational  Plan,  March  22,  1995,  overall  program 
success  will  ultimately  be  determined  by  the  impact  that  the  ECRC  program  has 
on  achieving  DoD's  goal  of  using  EC/EDI  to  conduct  business. 


ECRC  Focus 


Focus  of  ECRC  Program.  The  ECRCs  focus  on  helping  small-  and 
medium-sized  vendors  to  use  EC/EDI  technologies  to  conduct  business  when 
most  government  buying  activities  and  prime  contractors  are  not  using  EC/EDI. 
The  Deputy  Under  Secretary  of  Defense  (Logistics)  needs  to  focus  ECRC 
efforts  on  creating  a  business  case  to  clearly  and  realistically  demonstrate  to 
vendors  the  economic  advantages  of  using  EC/EDI.  Until  the  Government  and 
prime  contractors  convert  to  using  EC/EDI  technologies  to  do  business,  and 
require  small-  and  medium-sized  vendors  to  use  it  as  well,  efforts  to  teach 
vendors  how  to  use  EC/EDI  technologies  are  premature. 

Director,  DoD  Electronic  Commerce,  Efforts.  The  Director,  DoD  Electronic 
Commerce,  is  working  with  DoD  buying  activities  and  prime  contractors  to 
establish  incentives  for  small-  and  medium-sized  vendors  to  use  EC/EDI 
technologies.  To  accomplish  this,  the  Director  is  educating  the  major  DoD 
buying  activities  and  prime  contractors  about  EC  practices  and  methods, 
including  EC/EDI  technologies,  and  the  benefits  of  converting  to  EC.  Once 
these  buying  activities  convert  to  EC,  they  can  require  their  suppliers  to  convert 
to  using  EC/EDI  technologies  as  well.  That  should  encourage  those  wishing  to 
continue  to  do  business  with  DoD  and  its  prime  contractors  to  implement 
EC/EDI  technologies.  The  ECRCs  can  assist  the  Director,  DoD  Electronic 
Commerce,  by  educating  DoD  buying  activities  and  prime  contractors  to  help 
them  adopt  EC/EDI  teclmologies. 


ECRC  Core  Functions 


The  ECRC  program  established  the  following  core  functions  to  accomplish  its 
mission:  outreach,  education  and  training,  consultation,  and  technical  support. 

Outreach.  Regional  ECRCs  conduct  outreach  events  to  make  vendors  aware  of 
the  ECRCs  as  a  source  of  assistance  for  learning  about  and  using  EC/EDI 
technologies.  Outreach  events  include  trade  shows,  seminars,  conferences,  and 
technology  demonstrations. 

Education  and  Training.  Regional  ECRCs  provide  EC/EDI  training  courses 
to  vendors  and  Government  organizations.  Education  and  training  events 
include  exhibits  at  symposia  and  classroom-type  courses  on  EC/EDI  concepts 
and  use.  The  ECRCs  offer  12  courses,  4  of  which  address  EC/EDI  topics. 
According  to  ECRC  officials,  the  other  8  courses  pertain  to  non-EC/EDI  topics 
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such  as  technical  data  exchange,  document  management,  and  business  process 
improvements.  The  courses  are  offered  free  of  charge  or  at  a  nominal  fee  to 
cover  the  cost  of  materials. 

Consultation.  Regional  ECRCs  offer  a  consultation  service  to  help  vendors 
develop  an  understanding  of  their  needs  and  operations,  and  to  identify 
problems  that  can  be  resolved  by  EC/EDI  technologies. 

Technical  Support.  Regional  ECRCs  provide  up  to  80  hours  of  technical 
support  to  vendors  to  help  them  implement  or  increase  their  use  of  EC/EDI 
technologies.  Technical  support  events  include  discussions  with  customer 
organizations  about  how  to  obtain  value  added  network  services,  conducting 
needs  analyses,  or  recommending  which  hardware  and  software  vendors  should 
buy  to  implement  EC/EDI.  ECRC  officials  consider  telephone  conversations  to 
be  either  outreach  or  consultation. 


Effectiveness  of  ECRC  Core  Functions 


To  measure  the  effectiveness  of  core  functions,  regional  ECRCs  track  the 
number  of  core  functions  performed  (activity  metrics)  and  document  successful 
implementations  of  EC/EDI  technologies  (impact  metrics).  We  detemmed  that 
ECRC  core  functions  did  not  result  in  significant  implementation  or  increased 
use  of  EC/EDI  technologies  by  its  target  audience. 

ECRC  Activity  Metrics.  We  reviewed  ECRC  activity  metrics  and  contacted  a 
statistical  sample  of  ECRC  customer  organizations.  We  determined  that 
60  percent  of  ECRC  activity  metrics  were  reliable  for  audit  purposes  and  a 
projected  8.6  percent  of  ECRC  customers  received  effective  ECRC  support. 
See  Appendix  D  for  reliability  of  ECRC  activity  metrics. 

ECRC  Impact  Metrics.  We  also  reviewed  ECRC  impact  metrics  (su^ss 
stories)  and  contacted  a  judgmental  sample  of  ECRC  customer  organizations: 
13  percent  of  ECRC  impact  metrics  were  reliable  and  3  out  of  the  10  customers 
we  contacted  had  valid  "success  stories."  See  Appendix  E  for  reliability  of 
ECRC  impact  metrics. 

Plans  to  Redefine  Performance  Metrics.  DLA  officials  acknowledged  that 
existing  ECRC  performance  metrics  did  not  accurately  measure  program 
effectiveness.  According  to  die  DLA  ECRC  Program  Manager,  DLA  interns  to 
r^efine  the  performance  measures  and  base  them  on  program  outcomes.  For 
example,  DIA  proposes  to  measure  the  effectiveness  of  core  functions  bas^  on 
the  number  of  vendors  that  implement  EC/EDI  technologic  after  receiving 
ECRC  core  function  support,  rather  than  on  the  number  of  individuals  that 
receive  support.  The  Deputy  Under  Secretary  of  Defense  (Logistics)  needs  to 
establish  controls  to  ensure  tiiat  ECRC  program  performance  metrics  are 
accurately  tracked  and  reported.  Additionally,  the  Deputy  Under  Secretary  of 
Defense  (Logistics)  should  clearly  define  the  ECRC  target  audience  as 
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Government  buying  activities  and  their  trading  partners  and  focus  the  ECRCs  on 
helping  them  convert  to  EC/EDI  business  practices  and  methods  to  maximize 
the  benefits  of  ECRC  efforts. 

RedeHne  Core  Functions.  If  the  ECRC  program  is  to  be  of  significant  benefit, 
the  Deputy  Under  Secretary  of  Defense  (Logistics)  needs  to  redirect  the 
activities  of  the  ECRCs.  The  ECRCs  need  to  focus  on  helping  DoD  address 
solutions  to  the  current  impediments  to  the  use  of  EC/^I.  TTie  ECRCs  need  to 
focus  on  helping  DoD  buying  activities  and  prime  contractors  learn  how  to  use 
EC/EDI  technologies.  They  also  need  to  focus  on  helping  small-  and 
medium-sized  vendors  to  understand  the  business  case  and  economic  advantages 
to  using  these  technologies.  Once  the  ECRC  focus  is  redirected  and  their 
functions  are  redefined,  DoD  ECRC  program  managers  need  to  establish 
performance  measmes  with  which  to  periodically  evaluate  and  report  on 
program  effectiveness. 


Overlap  of  ECRC  Core  Functions 


ECRC  services  duplicate  the  EC/EDI  services  that  already  exist,  and  are  funded 
by  DoD,  other  Government  organizations,  and  private  industry.  In  addition  to 
funding  the  ECRCs,  the  Federal  Government  spends  about  $166  million 
annually  to  fund  109  Procurement  Technical  Assistance  Centers,  1,006  Small 
Business  Development  Centers  and  sub-centers,  and  60  Manufacturing 
Extension  Partnerships  (MEPs).  Like  the  ECRCs,  the  majority  of  these 
1,175  centers  provide  education  and  training,  including  how  to  implement  and 
use  EC/EDI  technologies,  at  little  or  no  cost  to  recipients. 

Procurement  Technical  Assistance  Cooperative  Agreement  Program.  In 
1985,  Congress  established  the  Procurement  Technical  Assistance  Cooperative 
Agreement  Program  to  provide  assistance  to  vendors  in  marketing  goods  and 
services  to  DoD  organizations,  and  designated  DLA  as  the  program  manager. 
The  Procurement  Technical  Assistance  Centers  provide  outreach,  counseling, 
marketing  opportunities,  and  EC/EDI  assistance  to  vendors.  The  FY  1997, 
National  Defense  Authorization  Act  re-emphasized  the  use  of  Procurement 
Technical  Assistance  Centers  to  implement  EC  initiatives  under  the  Federal 
Acquisition  Streamlining  Act  of  1994. 

Small  Business  Development  Centers.  The  Small  Business  Administration 
initiated  the  Small  Business  Development  Centers  as  a  pilot  project  in  FY  1977. 
In  FY  1980,  Congress  established  the  Small  Business  Development  Centers 
Program  to  provide  vendors  with  free  outreach,  management,  technical  support, 
training,  and  research  assistance.  Like  the  ECRCs,  the  Small  Business 
Development  Centers  provide  EC/EDI  training  to  vendors. 
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Manufacturing  Extension  Partnerships.  In  FY  1988,  Congress  established 
the  MEP  program  within  the  Department  of  Commerce,  National  Institute  of 
Standards  and  Technology,  to  provide  technical  assistance  to  vendors  to 
improve  their  operations  through  the  use  of  advanced  technologies.  The 
program  consists  of  60  centers  that  offer  a  wide  range  of  business  services. 
Like  the  ECRCs,  the  MEPs  assist  vendors  in  implementing  EC/EDI 
technologies.  In  responding  to  the  FY  1994  DoD  Appropriations  Act 
requirement  that  DoD  justify  continuing  the  MEP  program,  the  Director, 
DARPA,  stated  that  when  the  MEP  Program  matures  in  FY  1997,  it  will  assist 
vendors  in  adopting  new  technologies  to  meet  commercial  and  defense  needs. 
Therefore,  in  FY  1997,  the  ECRC  mission  to  provide  outreach,  education  and 
training,  consultation  and  technical  support  to  vendors  will  have  been  largely 
transferred  to  the  MEP  Program.  Figure  2  shows  the  EC  outreach  locations. 
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Figure  2.  Electronic  Commerce  Outreach  Sites 


Private  Industry  Sources  of  EC/EDI  Support.  We  also  identified  a  sample  of 
10  private  sector  sources  that  offer  EC/EDI  training  and  support  similar 
to  lhat  provided  by  the  ECRCs.  The  private  sources  include  individual 
contractors  and  national  associations  that  offer  assistance  at  various  fees. 
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Significance  of  Alternative  Sources.  The  services  provided  by  the  ECRCs 
duplicate  those  provided  by  other  organizations  that  are  already  in  place  and 
have  existing  relationships  with  Government  vendors.  Furthermore,  DoD 
officials  have  been  aware  of  the  overlapping  namre  of  the  ECRC  core  functions 
for  some  time.  In  1993,  Air  Force  officials  asked  the  Cleveland  ECRC  to 
examine  similarities  between  MEP  and  ECRC  core  functions,  and  identify  ways 
to  minimize  the  duplication.  The  subsequent  report,  "CSRC-Manufacturing 
Technology  Center  Study  Task  Order,"  December  26,  1993,  concluded  that  the 
scopes,  objectives,  activities,  and  technology  focus  of  the  two  programs  were 
duplicative,  and  recommended  that  the  programs  be  coordinated  to  reduce 
unnecessary  duplication. 

Opportunity  to  Coordinate  ECRC  Core  Functions.  The  Deputy  Under 
Secretary  of  Defense  (Logistics)  has  an  opportunity  to  eliminate  overlap 
between  the  various  organizations  providing  EC/EDI  outreach  and  training. 
The  Defense  Logistics  Agency,  as  program  manager  for  the  Procurement 
Technical  Assistance  Cooperative  Agreement  Program  and  ECRCs,  should 
analyze  the  functions  of  the  two  programs  and  coordinate  EC/EDI  efforts  to 
eliminate  duplication.  The  Defense  Logistics  Agency  should  also  establish  a 
Government-wide  electronic  EC/EDI  Integrated  Process  Team  to  coordinate 
ECRC  activities  with  other  Government  programs  having  similar  missions.  At 
a  minimum,  the  Integrated  Process  Team  should  attempt  to  include 
representatives  from  the  Office  of  Federal  Procurement  Policy,  the  Department 
of  Commerce,  the  Small  Business  Administration,  the  General  Services 
Administration,  the  DoD  Electronic  Commerce,  the  Defense  Finance  and 
Accounting  Service,  and  the  Military  Departments. 


Technology  Hub  Initiatives 


DARPA  officials  considered  the  hub  concept  to  be  flawed  and  hub  initiatives 
generally  not  of  significant  value  to  DoD.  CTC  asserted  the  need  for  the  hub, 
and  in  FY  1994,  Congress  established  the  hub  under  CTC,  and  collocated  it 
with  the  Johnstown  ECRC  and  team  integrator.  The  purpose  of  the  hub  was  to 
identify,  or  develop,  and  deploy  new  or  existing  technologies  to  resolve 
problems  preventing  Government  organizations  and  vendors  from  using 
EC/EDI,  and  to  serve  as  a  source  of  EC/EDI  information  for  regional  ECRCs. 
In  addition  to  identifying  the  requirement  for  the  hub,  CTC  determined  its 
mission,  identified  initiatives  to  accomplish  the  mission,  and  proposed  the 
initiatives  to  the  DoD  ECRC  program  office  for  approval.  We  conducted  a 
technical  review  of  hub  initiatives  and,  like  DARPA,  foimd  none  of  the 
initiatives  to  be  of  significant  value  to  DoD.  The  combined  budget  for  the 
15  initiatives  was  $15.8  million  for  FYs  1994  through  1995. 

Hub  Initiatives.  The  hub  undertook  seven  initiatives  in  FY  1995  and  eight 
initiatives  in  FY  1996.  All  but  one  of  the  15  initiatives  were  self-initiated  by 
CTC.  The  remaining  initiative  was  recommended  by  the  ECRC  Contracting 
Officer's  Technical  Representative.  According  to  the  DARPA  ECRC  program 
manager,  when  DoD  did  not  approve  the  CTC-proposed  initiatives.  Congress 
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mandated  them.  For  example,  the  first  four  hub  initiatives  were  directed  in  the 
House  of  Representatives,  "Report  of  the  Committee  on  Appropriations,"  which 
accompanied  the  FY  1994  DoD  Appropriations  Act. 

Analysis  of  Hub  Initiatives.  DARPA  and  the  Office  of  Inspector  General, 
DoD,  independently  concluded  that  the  hub  initiatives  were  of  little  or  no  value 
in  meeting  DoD  EC/EDI  goals. 

DARPA  Analysis  of  Hub  Initiatives.  In  a  February,  1996,  brief  to 
Inspector  General,  DoD,  auditors,  DARPA  ECRC  program  officials  opined  the 
hub  concept  is  flawed  and,  generally,  the  hub  initiatives  were  not  of  significant 
value  to  DoD.  Specifically,  DARPA  considered  the  hub  concept  flawed  as  a 
means  of  technology  development,  DARPA  ECRC  program  officials  thought 
there  would  be  no  need  for  the  hub  if  normal  competitive  procedures  were  used 
to  pursue  development  (a  more  effective  procedure  for  this  purpose),  and  the 
other  hub  functions  were  folded  into  team  integrator  and  core  function  tasks. 
DARPA  officials  thought  that,  where  hub  activities  were  worth  pursuing,  there 
were  better  ways  to  accomplish  them.  Also,  even  though  the  focus  of  the 
ECRCs  was  redirected  from  CALS  to  EC/EDI  in  FY  1994,  8  of  the  15 
FYs  1995  and  1996  hub  initiatives  focused  on  CALS  issues. 

Audit  Analysis  of  Hub  Initiatives.  We  performed  a  technical  review 
of  the  15  hub  initiatives.  The  pui^ose  of  the  review  was  to  determine  if  the 
initiatives  had  merit  and  met  the  objectives  of  the  hub.  The  objective  of  the  hub 
was  to  identify  new  or  existing  technologies  to  resolve  vendor  and  Government 
organizations'  impediments  to  implementing  EC/EDI .  The  technical  review 
concluded  that  none  of  the  initiatives  provided  technologies  that  were  not 
already  available  within  the  marketplace,  and  that  funds  expended  on  the  hub 
could  be  put  to  better  use.  In  addition,  at  least  3  of  the  initiatives  were 
functions  tiiat  should  have  been  conducted  directly  under  the  ECRCs,  and  not 
the  hub.  The  following  are  examples  of  questionable  initiatives.  See 
Appendix  F  for  a  discussion  of  the  remaining  hub  initiatives. 

CALS  Exposition.  The  hub  spent  $594,574  to  plan  and  develop  ECRC 
exhibits  for  FYs  1994  and  1995  CALS  expositions.  In  addition,  the  hub 
budgeted  $290,265  for  the  FY  1996  CALS  exposition.  These  projects  were 
self-initiated.  The  exhibits  included  a  kiosk,  handouts  from  each  of  the  regional 
ECRCs,  and  a  computerized  presentation  about  the  ECRCs.  Such  an  initiative 
should  have  been  funded  under  the  ECRC  program  directly.  Further,  this 
initiative  did  not  achieve  the  hub  objective  of  identifying  technologies  to  assist 
in  implementing  EC/EDI .  Rather,  the  initiative  served  as  a  marketing  tool  for 
the  ECRCs. 

Collaborative  Application  Protocol  Development  Initiative.  The  hub 
spent  $1.1  million  in  FY  1995  to  identify  methods  to  expedite  development  of 
application  protocols  for  the  "Standards  for  the  Exchange  of  Product  Model 
Data  (STEP). "  The  House  Appropriations  Committee,  in  its  report  on  the  1994 
DoD  Appropriations  Bill,  directed  the  establishment  of  this  initiative.  The 
purpose  of  STEP  is  to  provide  standard  formats  for  representing  and  exchanging 
information  about  any  product  during  all  phases  of  the  product's  life  cycle, 
including  the  transfer  of  technical  data  between  organizations  using  different 
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software  packages,  hardware,  and  operating  systems.  According  to  DoD 
program  officials,  this  initiative  did  not  contribute  to  the  ECRC  program 
because  it  focused  on  manufacturing  processes  and  issues  rather  than  on  the 
EC/EDI  needs  of  vendors.  Further,  the  Inspector  General,  DoD,  technical 
review  concluded  that  this  initiative  duplicated  National  Institute  of  Standards 
and  Technologies  efforts  and  attempted  to  develop  a  standard  that  industry  has 
not  seen  fit  to  create  on  its  own.  The  hub  ended  this  initiative  in  FY  1995 
without  producing  anything  beneficial  to  vendors  or  DoD. 

Value  of  Hub.  The  stated  purpose  of  the  hub  was  to  identify  new  or  existing 
technologies  that  could  resolve  problems  Government  organizations  and  vendors 
were  having  in  implementing  EC/EDI  technologies,  and  to  act  as  a  source  of 
EC/EDI  information  for  the  regional  ECRCs.  However,  based  upon  the 
technical  assessment  of  the  initiatives,  we  question  whether  DoD  funds  are 
being  used  efficiently.  The  initiatives  have  not  identified  solutions  or  resolved 
problems.  In  addition,  the  initiatives  were  not  based  on  vendor  or  Government 
EC/EDI  needs. 

DoD  expended  approximately  $15.8  million  authorized  for  FYs  1994  and  1995 
to  fund  hub  initiatives.  Additional  funds  will  be  expended  to  continue  these 
initiatives  for  FYs  1996  through  1998,  however,  hub  funding  has  not  been 
broken  out.  Our  analysis  supported  DARPA's  conclusion  that  the  hub 
initiatives  are  not  of  significant  value.  In  our  opinion,  the  hub  has  not 
demonstrated  that  it  has  accomplished  its  objectives  and  provided  useful  tools  to 
vendors  or  Government  organizations  to  solve  EC/EDI  problems.  Therefore, 
we  believe  the  Deputy  Under  Secretary  of  Defense  (Logistics)  should  seek 
authorization  to  eliminate  the  congressionally-directed  ECRC  technology  hub. 


ECRC  Program  Infrastructure 


Ten  Government  and  private  industry  organizations  as  well  as  subcontractors 
participate  in  ECRC  program  management,  oversight,  and  execution.  Four  of 
the  ten  Government  and  private  industry  organizations  involved  in  program 
infrastructure  used  almost  34  percent,  or  $25.7  million  of  the  $76.7  million,  of 
the  FYs  1995  and  1996  ECRC  budget  for  program  management  and  ECRC  non¬ 
core  functions.  The  four  organizations  and  the  related  amounts  were  as  follows: 
Concurrent  Technologies  Corporation  used  $15.8  million  to  operate  the  hub; 
DARPA  used  $7.6  million  to  fond  technology  projects  and  acquire  planning  and 
oversight  support;  and  the  two  team  integrators  used  $2.3  million  collectively 
to  plan,  oversee,  and  coordinate  regional  ECRC  activities.  These  expenditures 
were  largely  unnecessary,  added  little  or  no  benefit  to  the  ECRC  program,  and 
detracted  from  the  regional  ECRCs  ability  to  provide  core  function  support. 

DLA  Plans  Program  Infrastructure.  DLA  officials  plan  to  initially  continue 
the  multi-layered  ECRC  program  management  structure,  including  the  Institute 
for  Defense  Analyses,  Fleet  Industrial  Supply  Center,  Production  Technology, 
Incorporated,  and  the  team  integrators.  DLA  officials  plan  to  review  program 
management  and  oversight  for  possible  restructuring  at  a  later  date.  We  believe 
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the  functions  performed  by  these  organizations  would  be  better  accomplished  by 
the  DLA  ECRC  program  office.  Costs  associated  with  the  necessary  program 
management  and  oversight  functions  total  only  $1.8  million  for  FYs  1994  and 
1995,  including  the  Contractor  Technical  Representative  function  accomplished 
by  the  Naval  Industrial  Resources  Support  Activity. 

Shift  of  Organizational  Control.  The  multi-layered  ECRC  program 
infrastructure  resulted  from  congressional  direction  and  DARPA  delegation  of 
program  management  and  oversight  responsibilities.  Congress  shifted 
management  responsibility  among  DoD  organizations  and  designated  ECRC 
locations,  contractors,  and  projects.  Congressional  direction  also  provided  a 
significant  level  of  autonomy  to  contractors.  The  shifting  of  organizational 
control  and  the  significant  level  of  contractor  autonomy  inhibited  DoD's  ability 
to  manage  the  program. 

For  purposes  of  discussing  the  efficiency  and  cost-effectiveness  of  ECRC 
program  infrastructure,  we  focused  on  FYs  1995  and  1996,  a  period  in  which 
DARPA  managed  the  program,  and  related  budgets  totaling  $76.7  million.  A 
brief  discussion  of  the  responsibilities  of  each  component  of  the  infrastructure  is 
included  in  Appendix  C.  The  chart  in  Appendix  C  presents  a  breakout  of  the 
$76.7  million  in  funding  for  FYs  1994  through  1995  as  allocated  across  the 
ECRC  program  infrastructure. 


Constraints  on  ECRC  Program  Flexibility 


We  believe  that  this  program  can  be  made  more  effective  if  DoD  is  allowed  to 
exercise  more  discretion  in  its  management,  including  the  authority  to  establish 
and  close  ECRCs,  to  select  contractors  through  the  competitive  process,  and  to 
determine  which  Departmental  agency  will  have  overall  program  responsibility. 
As  this  program  has  developed,  many  of  these  normal  management  decisions 
have  been  made  by  Congress.  For  example.  Congress: 

0  shifted  ECRC  program  management  responsibility  from  Air  Force  to 
DARPA,  to  DLA,  to  the  Deputy  Under  Secretary  of  Defense  (Logistics); 

o  designated  CTC  and  CAMP  as  ECRC  sole-source  contractors  and 
identified  their  responsibilities; 

o  designated  the  locations  of  the  regional  ECRCs,  including 
reestablishing  the  Orange,  Texas  ECRC  location; 

o  directed  that  CTC  establish  and  maintain  a  technology  hub  even 
though  there  was  no  clearly  defined  need  for  a  hub  function; 

o  designated  at  least  4  of  the  15  initiatives  performed  by  the  hub,  the 
first  of  which  was  to  identify  a  need  for  the  hub;  and 
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0  directed  the  establishment  of  five  new  ECRCs  in  FY  1997  even 
though  the  contribution  of  the  existing  ECRCs  to  acquisition  reform  efforts  was 
limited. 

Those  constraints  reflected  congressional  dissatisfaction  with  DoD 
administration  of  the  program. 

Air  Force  Management  of  the  ECRC  Program.  In  FY  1991,  Congress 
designated  the  Air  Force  as  the  ECRC  program  manager.  To  respond  to 
congressional  interest  and  quickly  establish  the  ECRC  program,  the  Air  Force 
prepared  an  initial  contract  with  CTC  that  contained  a  broad  statement  of  work 
with  few  deliverables,  and  limited  Air  Force  involvement  in  program 
management.  When  Air  Force  revised  the  CTC  contract  statement  of  work  to 
clearly  define  Air  Force  and  CTC  roles  and  responsibilities,  CTC  vigorously 
resisted  the  changes.  Specifically,  according  to  Air  Force  officials,  CTC 
officials  stated  that  the  "Air  Force  was  to  fond  CTC,"  and  "should  not  attempt 
to  manage  the  program."  Further,  CTC  officials  stated  that  "CTC  was  not 
interested  in  being  the  Air  Force's  support  contractor  or  hired  help,  and  that 
CTC  would  provide  Air  Force  with  the  results  of  CTC-selected  initiatives." 

Congress  directed  the  Air  Force  to  award  sole-source  follow-on  contracts  to 
CTC  and  I-Net,  Incorporated.  In  an  October  7,  1993,  memorandum  to  the 
Senate  Appropriations  Committee,  the  Deputy  Assistant  Secretary  of  the  Air 
Force,  Communications,  Computers,  and  Support  Systems,  stated  that,  if  the 
Air  Force  was  not  allowed  to  control  and  manage  the  ECRC  program  and 
exercise  authority  over  ECRC  funding,  they  would  prefer  that  ECRC  fonding  be 
given  to  the  Office  of  the  Secretary  of  Defense  instead  of  the  Air  Force. 

Congress  transferred  ECRC  program  management  responsibility  from  the  Air 
Force  to  ARPA  (now  DARPA)  in  the  FY  1994  DoD  Appropriations  Act.  In 
House  Report  No.  103-254,  accompanying  the  legislation,  the  conunittee  stated 
that  "the  Air  Force  repeatedly  altered  congressional  intent  in  the  implementation 
of  the  ECRC  program. "  The  Report  forther  stated  that  the  Air  Force  refused  to 
establish  the  ECRCs  as  the  tri-service  CALS  standards  and  technologies 
development,  deployment,  training,  and  education  hub  (designate  CTC  as  the 
center  of  excellence  for  electronic  technology).  Also,  the  Air  Force  had  not 
established  an  ECRC  in  Palestine,  Texas,  as  directed  by  Congress.  Instead  they 
had  redirected  congressionally-earmarked  ECRC  funds  to  establish  and  operate 
an  activity  to  convert  existing  (legacy)  weapons  systems  documents  into 
electronic  form.  The  Air  Force  considered  legacy  data  conversion  more 
beneficial  to  achieving  DoD's  EC/EDI  objectives  foan  establishing  another 
regional  ECRC  to  train  organizations  that  may  or  may  not  use  EC/EDI  to  do 
business  with  the  Government  in  the  future. 

Prior  to  establishment  of  the  ECRCs,  the  Air  Force  CALS  budget  averaged 
about  $20  million  a  year.  Subsequent  to  establishment  of  the  ECRC  program, 
the  annual  Air  Force  CALS  budget  grew  to  about  $50  million.  When  the 
ECRC  program  was  moved  to  DARPA,  Congress  reduced  the  Air  Force  CALS 
budget  from  $50  million  to  $2.3  million,  effectively  eliminating  the  Air  Force 
CALS  program. 
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DARPA  Management  of  the  ECRC  Program.  Upon  receiving  the  ECRC 
program  from  Air  Force  in  FY  1994,  DARPA  reduced  funding  for  the  hub  by 
about  $2  million  and  tried  to  shift  the  ECRC  focus  firom  CALS  initiatives 
toward  EC/EDI  functions.  Although  the  DARPA  ECRC  program  manager 
believed  hub  activities  could  be  better  performed  by  competitively  selected 
contractors,  congressional  direction  explicitly  precluded  this.  Also,  because  the 
hub  was  congressionally  mandated,  DARPA  could  not  abolish  it  even  though 
they  considered  the  majority  of  the  hub  initiatives  to  be  of  little  value. 

DLA  Management  of  the  ECRC  Program.  In  the  FY  1996  DoD 
Appropriations  Act,  Congress  transferred  ECRC  program  management  to  DLA 
in  FY  1997,  stating  that  the  move  to  DLA  will  better  integrate  the  EC  needs  of 
DoD  and  its  suppliers.  Subsequently,  Congress  directed  in  the  Conference 
Report  accompanying  the  Omnibus  Consolidated  Appropriations  Act  for 
FY  1997,  that  the  ECRC  program  be  moved  under  the  Deputy  Under  Secretary 
of  Defense  (Logistics).  However,  the  Deputy  Under  Secretary  of  Defense 
(Logistics)  does  not  have  the  staff  to  oversee  the  ECRC  Program.  As  a  result, 
the  Deputy  Under  Secretary  of  Defense  (Logistics)  delegated  program 
management  responsibilities  to  DLA. 


Contracting  Methodology 


The  contracting  methods  DoD  used  to  acquire  ECRC  support  were  not  in  the 
Government  best  interest  because  they  did  not  provide  incentive  for  contractors 
to  use  efficient,  cost-effective  methods  and  practices.  Also,  the  contracting 
method  used  exposed  DoD  to  the  greatest  degree  of  risk  and  resulted  in  CTC 
and  CAMP  being  paid  fees  totaling  approximately  $900,000  from  FYs  1994 
through  1996  that  were  not  adequately  justified. 

Contract  Type  Selection.  Federal  Acquisition  Regulation,  Subpart 
16.103,  "Selecting  Contract  Types,"  Section  103,  states  that  the  type  of  contract 
used  should  result  in  reasonable  contractor  risk  while  providing  the  contractor 
with  the  greatest  incentive  for  efficient  and  economical  performance. 
Cost-reimbursement-type  contracts  such  as  cost-plus-award-fee  and 
cost-plus-fixed-fee  are  appropriate,  when  uncertainties  in  contract  performance 
do  not  permit  costs  to  be  estimated  with  sufficient  accuracy  to  use  fixed-price 
contracts.  Cost-reimbursement-type  contracts  should  be  used  only  when 
Government  oversight  ensures  efficient  contract  monitoring  and  effective  cost 
control.  Contracting  officials  should  reevaluate  the  appropriateness  of  the 
contract  type  used  as  a  program  matures,  and  change  the  contract  type  when 
experience  provides  a  basis  for  firmer  pricing. 

Contract  Methods  Used  to  Acquire  ECRC  Support.  In  FYs  1991 
through  1993,  Congress  directed  the  Air  Force  to  award  sole-source  contracts  to 
CTC  and  I-Net,  Incorporated,  to  operate  the  two  original  regional  ECRCs.  The 
Air  Force  awarded  cost-plus-award-fee  contracts  to  operate  five  additional 
ECRC  sites,  and  justified  the  use  of  this  contract  type  by  stating  that  the  nature 
of  the  work  involved  uncertain  research,  development,  test,  and  evaluation 
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efforts.  However,  the  effort  required  to  perform  the  ECRC  core  functions  of 
outreach,  education  and  training,  consultation,  and  technical  support  were  not 
uncertain.  The  nature  of  the  contracted  work  did  not  involve  research, 
development,  test,  and  evaluation.  Cost-plus-award-fee  contracts  entitled  the 
contractors  to  be  reimbursed  for  all  allowable  costs  and  to  receive  an  award  fee 
based  on  contractor  performance. 

When  the  ECRC  program  transferred  to  DARPA,  DARPA  continued  using 
cost-reimbursement-type  contracts  but  awarded  cost-plus-fixed-fee  contracts 
rather  than  cost-plus-award-fee  contracts.  DARPA  justified  using 
cost-plus-fixed-fee  contracts,  stating  that  the  complex  technical  difficulties  in 
performing  ECRC  support  services  could  not  be  predicted  in  advance.  The 
DARPA  contracting  officer  further  justified  paying  contractors  a  fixed-fee, 
stating  that  the  award  fees  did  not  provide  incentive  to  ECRC  contractors 
because  they  almost  automatically  received  100  percent  of  the  award  fees 
anyway.  Therefore,  it  was  not  worth  the  added  cost  to  the  Government  to 
administer  award-fee-type  contracts. 

Opportunity  to  Re-evaluate  Contract  Type.  The  Deputy  Under  Secretary  of 
Defense  (Logistics)  designated  DLA  as  the  ECRC  program  manager,  beginning 
in  FY  1997.  As  the  designated  ECRC  program  manager,  DLA  officials  plan  to 
continue  using  cost-plus-fixed-fee  contracts  because  they  believe  this  contract 
type  is  more  appropriate  for  programs  funded  within  the  research,  development, 
test,  and  evaluation  appropriation.  However,  the  program  manager  agreed  with 
the  IG,  DoD,  recommendation  to  reconsider  the  use  of  cost-plus-fixed-fee 
contracts  because  the  ECRC  program  does  not  involve  research,  development, 
test,  or  evaluation  efforts.  Also,  because  the  ECRCs  have  been  in  operation  for 
about  6  years,  DLA  officials  stated  that  they  have  sufficient  information  to 
clearly  define  ECRC  contract  statements  of  work.  In  the  past,  poorly  defined 
statements  of  work  allowed  contractors  too  much  autonomy  and  provided  little 
incentive  to  the  contractors.  Therefore,  the  Deputy  Under  Secretary  of  Defense 
(Logistics)  and  DLA  should  consider  changing  the  contract  type  from 
cost-plus-fixed-fee  contracts  to  firm-fixed-price  contracts.  Firm  fixed-price 
contracts  provide  maximum  incentive  for  the  contractor  to  control  cost  and 
perform  effectively  while  minimizing  the  administrative  burden  on  the 
contracting  parties. 

Contract  Fees  and  Nonprofit  Organizations.  Contract  fees  are  negotiated 
amounts  paid  to  contractors  in  addition  to  all  reimbursable  contract  costs.  Fees 
are  paid  to  nonprofit  organizations  to  provide  operating  capital  and  cover  non¬ 
reimbursable  expenses  that  are  ordinary  and  necessary  to  the  successful 
operation  of  the  organization.  Further,  Office  of  Management  and  Budget 
Circular  A-122,  "Cost  Principles  for  Nonprofit  Organizations,"  limits  the  use  of 
fees  to  maintaining,  improving,  or  expanding  operations.  Also,  the  Defense 
Federal  Acquisition  Regulation  Supplement  215.972,  "Modified  Weighted 
Guidelines  for  Nonprofit  Organizations,"  215.972(d),  states  that,  where  the 
payment  of  fees  is  appropriate,  contracting  officers  should  compute  the  fee  rate 
using  the  weighted  guidelines  method  in  Defense  Federal  Acquisition  Regulation 
Supplement  215.971-l(a),  which  focuses  on  the  following  factors:  performance 
risks;  contract  type  risks;  and  facilities  capital  employed  (i.e.,  buildings  and 
office  furniture). 
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Fees  Paid  to  ECRC  Contractors.  DARPA  paid  CTC  and  CAMP  contract  fees 
totaling  approximately  $900,000  without  requiring  the  contractors  to  justify  the 
fees  to  maintain,  improve,  or  expand  their  ECRC  operations  or  to  provide 
evidence  that  fees  were  used  for  authorized  purposes.  Furthermore,  in 
computing  the  fee  rate,  DoD  contracting  officials  relied  on  what  was  customary 
for  research,  development,  test,  and  evaluation  contracts  rather  than  the 
weighted  guidelines.  For  CTC  and  CAMP,  performance  and  contract  risks 
were  minimal  because  of  the  nature  of  the  work  involved  and  the  contract  type 
used.  Also,  facilities  capital  employed  was  not  a  factor  for  CTC  and  CAMP 
because  their  furnishings  and  equipment  were  provided  by  the  Government,  and 
their  office  spaces  were  leased. 

Opportunity  to  Evaluate  Contract  Fees.  The  Deputy  Under  Secretary  of 
Defense  (Logistics)  should  require  that  DLA,  as  the  designated  ECRC  program 
manager,  review  the  fees  paid  to  ECRC  contractors  and  ensure  that  the  fees  are 
appropriate.  To  accomplish  this,  contracting  officials  should  include  a  use  of 
fee  clause  in  ECRC  contracts  that  requires  contractors  to  justify  their  need  for 
fees.  As  a  minimum,  annual  fee  requests  should  include:  a  description  of  each 
fee  expense  including  an  explanation  of  how  the  expenditure  benefits  the  ECRC 
program,  and  an  explanation  of  why  the  expense  cannot  be  charged  under 
existing  cost  guidelines;  and  a  report  on  the  use  of  prior  year's  fees.  This 
information  is  needed  for  DoD  contracting  officials  to  determine  whether  fees 
are  appropriate  and  used  in  accordance  with  governing  regulations.  Also,  the 
Deputy  Under  Secretary  of  Defense  (Logistics)  and  DLA  officials  should 
analyze  the  need  for  fees  on  an  annual  basis  and  reduce  fee  awards  for 
unexpended  prior  year's  fees. 


Cost  Effectiveness  of  ECRCs 


The  current  emphasis  within  DoD  and  the  Government  is  to  re-evaluate 
processes  and  the  way  we  do  business,  and  to  streamline  wherever  possible  to 
be  more  efficient  and  cost-effective.  Further,  the  expanded  use  of  i^ormation 
technology  capabilities  throughout  the  Government  and  the  private  sector  is 
helping  push  the  implementation  of  EC/EDI  into  this  infrastructure.  As  a 
result,  if  DoD  and  tiie  Government  want  to  assist  vendors  and  procurement 
offices  with  this  implementation  effort,  they  need  to  do  it  in  a  way  that 
efficiently  achieves  the  goal  and  is  cost-effective. 

The  following  provides  some  perspective  on  the  costs  incurred  by  the  ECRCs  to 
accomplish  Aeir  mission  to  assist  vendors  and  Government  organizations  in 
implementing  and  using  EC/EDI  technologies.  DoD  obligated  approximately 
$81.5  million  to  fund  the  11  regional  ECRCs  and  the  hub  during  FY  1994 
through  the  first  quarter  of  FY  1996.  During  that  period,  the  ECRCs  provided 
support  to  12,764  identifiable  organizations.  Based  on  our  sample,  we  project 
that  about  1,102  of  these  organizations  implemented  or  increased  their  use  of 
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EC/EDI  technologies  as  a  result  of  ECRC  core  function  support.  Averaging  the 
$81.5  million  over  the  1,102  organizations,  it  cost  DoD  about  $74,000  on  the 
average  for  each  organization  that  implemented  or  increased  its  use  of  EC/EDI 
technologies. 

Looking  at  ECRC  costs  another  way,  there  are  approximately  208,000  vendors 
throughout  the  United  States  that  conduct  business  with  the  Federal 
Government,  and  1,400  DoD  procurement  activities  that  need  to  use  EC/EDI 
technologies  to  do  business.  If  DoD  used  the  1,175  Procurement  Technical 
Assistance  Centers,  Small  Business  Development  Centers,  and  MEPs  to  train 
the  209,400  vendor  and  DoD  organizations,  they  would  each  have  to  provide 
training  to  only  178  organizations. 


Deputy  Under  Secretary  of  Defense  (Logistics)  and  Defense 
Logistics  Agency  Redirection  of  the  ECRC  Program 


As  the  designated  ECRC  program  manager,  DLA  has  the  opportunity  to 
evaluate  and  make  the  ECRC  program  more  effective.  Further,  since  DLA  has 
responsibility  for  the  Procurement  Technical  Assistance  Cooperative  Agreement 
program  as  well  as  the  ECRC  program,  they  need  to  coordinate  the  efforts  of 
the  two  programs  and  eliminate  duplication  of  effort.  DLA  ECRC  and 
Procurement  Technical  Assistance  Cooperative  Agreement  program  officials 
already  agreed  that  this  is  appropriate  action  and  are  forming  an  Integrate:d 
Process  Team  to  minimize  program  overlap.  In  addition,  since  there  is 
duplication  among  EC  efforts  performed  by  ECRCs,  Procurement  Technical 
Assistance  Centers,  MEPs,  and  Small  Business  Development  Centers,  the 
Deputy  Under  Secretary  of  Defense  (Logistics)  should  encourage  the 
establishment  of  a  Government-wide  Integrated  Process  Team  to  coordinate 
these  activities  to  eliminate  duplication  and  provide  services  in  the  most  efficient 
and  cost-effective  manner. 

Refocusing  ECRC  Efforts.  DLA,  in  anticipation  of  receiving  the  program, 
recognized  the  need  to  refocus  ECRC  efforts.  Some  options  being  considered 
by  DLA  to  redirect  the  ECRC  program  included  tasking  the  ECRCs  to  provide 
more  hands-on  EC/EDI  training,  and  channeling  ECRC  assistance  to  key  DoD 
vendors.  The  Deputy  Under  Secretary  of  Defense  (Logistics),  since  being 
delegated  responsibility  for  the  ECRC  program,  supports  DLA  efforts  to 
redirect  the  program. 

Provide  Hands-on  EC/EDI  Training.  DLA  officials  attended  some 
training  events  provided  by  the  ECRCs  in  anticipation  of  taking  over  the 
program  and  aclmowledged  that  education  and  training  core  functions  were  not 
effective  in  getting  vendors  to  use  EC/EDI  technologies  to  do  business  with 
DoD.  The  training  was  not  effective  because  the  ECRCs  focused  too  much  on 
theory  and  not  enough  on  how  to  use  EC/EDI  technologies.  Also,  only  4  of  the 
12  ECRC  courses  focus  on  EC/EDI .  As  the  designated  program  manager,  DLA 
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officials  plan  to  have  the  ECRCs  revise  their  courses  to  provide  hands-on 
training  to  teach  vendors  how  to  use  EC/EDI  technologies  to  conduct 
acquisition  transactions. 

Channeling  ECRC  Assistance.  As  the  designated  program  manager, 
DLA  acknowledged  that  ECRC  efforts  were  not  focused  on  the  target  audience. 
To  alleviate  this,  DLA  plans  to  have  the  ECRCs  identify  and  assist  the  small- 
and  medium-sized  DoD  vendors  who  are  not  using  EC/EDI  technology. 
Initially,  DLA  will  provide  the  ECRCs  a  data  base  of  DLA  vendors  and  plans  to 
provide  them  with  similar  data  bases  for  the  Military  Departments  to  ensure  that 
future  ECRC  efforts  are  focused  on  appropriate  target  audiences. 

Other  Potential  ECRC  Tasks.  As  the  designated  program  manager,  DLA 
officials  might  also  consider  tasking  the  ECRCs  to  perform  other  functions  that 
do  not  duplicate  the  outreach  and  training  efforts  of  Procurement  Technical 
Assistance  Centers,  MEPs,  and  Small  Business  Development  Centers  already  in 
place  and  working  with  vendors  on  a  daily  basis.  For  example,  DLA  could  task 
the  ECRCs  to  help  the  Director,  DoD  Electronic  Commerce,  with  efforts  to 
educate  DoD  buying  activities  and  prime  contractors  in  using  EC/EDI 
technologies.  In  addition,  the  ECRCs  could  help  small-  and  medium-sized 
vendors  understand  the  business  case  and  economic  advantages  of  using  EC/EDI 
technologies,  and  help  them  to  register  in  the  Central  Contractor  Registration 
System.  DLA  could  also  task  the  ECRCs  to  work  directly  with  DoD  and  other 
Government  agency  Small  and  Disadvantaged  Business  Utilization 
representatives  to  identify  and  channel  assistance  to  vendors  who  do  business 
with  the  Government. 


Establishment  of  Additional  ECRCs 


FY  1997  Congressional  Guidance  on  ECRCs  in  the  Defense  Appropriations 
Act.  The  Conference  Report  accompanying  the  "Omnibus  Consolidated 
Appropriations  Act,  1997,"  provided  the  fcRC  program  an  additional 
$15  million  specifically  for  the  establishment  and  operation  of  five  additional 
regional  ECRCs.  The  final  decisions  on  the  establishment  of  the  five  ECRCs 
must  be  made  within  90  days  of  the  enactment.  The  Report  also  directed  DoD 
to  submit  a  plan  by  March  15,  1997,  showing  how  regional  ECRCs  can  be 
increasingly  supported  by  non-governmental  activities  and  be  entirely  self- 
sustaining  in  5  years.  This  plan  should  also  identify  additional  areas  in  DoD 
that  could  benefit  from  the  use  of  electronic  commerce  technology  and  explain 
how  ECRCs  can  be  used  to  meet  these  emerging  requirements. 

ECRC  Expansion.  Based  on  the  results  of  this  audit,  we  do  not  believe  it  is 
advisable  to  expand  the  existing  network  of  ECRCs  at  this  time.  The  hub  had 
not  produced  any  solutions  to  vendor  problems.  The  current  ECRC  mission 
was  only  8.6  percent  effective  in  promoting  the  use  of  EC/EDI  technologies  in 
support  of  acquisition  streamlining  initiatives.  Furthermore,  the  Deputy  Under 
Secretary  of  Defense  (Logistics)  and  DLA  officials  have  not  had  adequate  time 
to  assess  or  redefine  Ae  mission  of  existing  ECRCs.  DLA  plans  to  spend  the 
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first  year  as  the  designated  program  manager  evaluating  how  to  best  use  ECRC 
resources.  Consistent  with  the  Conference  Committee  guidance,  the  Deputy 
Under  Secretary  of  Defense  (Logistics)  and  DLA  officials  also  have  to  devise  a 
plan  to  make  the  ECRC  program  entirely  self-supporting  in  5  years.  The 
Deputy  Under  Secretary  of  Defense  (Logistics)  and  DLA  need  time  to 
accomplish  these  objectives.  Therefore,  we  believe  that  the  Deputy  Under 
Secretary  of  Defense  (Logistics)  should  pursue  authorization  to  delay 
establishment  of  die  five  new  ECRCs  to  allow  DLA  time  to  redefine  die  ECRC 
mission  consistent  with  the  new  congressional  direction  rather  than  have  them 
expand  the  current,  ineffective  ECRC  program. 


Recommendatioiis  for  Corrective  Action 


1.  We  recommend  that  the  Deputy  Under  Secretary  of  Defense  (Logistics): 

a.  Pursue  authorization  to  delay  establishment  of  the  five  new  electronic 
commerce  resource  centers  to  allow  the  Director,  Defense  Logistics,  time  to 
evaluate  how  to  best  use  existing  resources,  in  accordance  widi  the  National 
Defense  Authorization  Act  for  FY  1997. 

b.  Seek  authorization  to  eliminate  the  congressionally-directed 
technology  hub, 

2.  We  recommend  that  the  Deputy  Under  Secretary  of  Defense  (Logistics) 
require  the  Director,  Defense  Logistics  Agency,  to: 

a.  Prioritize  electronic  commerce  resource  center  tasking  to  focus  on 
getting  DoD  buying  activities  and  prime  contractors  to  use  electronic 
commerce/electronic  data  interchange  technologies  to  do  business,  and  helping 
small-  and  medium-sized  businesses  understand  the  business  case  and  economic 
advantages  of  using  them  as  well. 

b.  Redefine  the  electronic  commerce  resource  center  target  audience  and 
require  regional  electronic  commerce  resource  centers  to  focus  support  on 
Government  buying  activities  and  their  suppliers. 
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c.  Consistent  with  Conference  Committee  guidance  in  the  Onmibus 
Consolidated  Appropriations  Act,  1997,  identify  areas  that  could  benefit  from 
electronic  conunerce  technology  and  explam  how  the  electronic  commerce 
resource  centers  can  be  used  to  meet  these  requirements.  Specifically,  consider 
tasking  the  electronic  commerce  resource  centers  to: 

(1)  Coordinate  with  the  DoD  procurement  activities  and  their 
Small  and  Disadvantaged  Business  Utilization  representatives  to  identify 
primary  vendors  who  are  not  using  electronic  commerce/electronic  data 
interchange. 


(2)  Help  those  vendors  identified  in  c.(l)  implement  electronic 
commerce/electronic  data  interchange  technologies. 

(3)  Help  vendors  register  in  the  Central  Contractor  Registration 

System. 


(4)  Help  the  Director,  DoD  Electronic  Commerce,  train  DoD 
buying  activities  and  prime  contractor  personnel  to  use  electronic 
commerce/electronic  data  interchange  technologies. 

d.  Establish,  and  incorporate  in  electronic  commerce  resource  center 
contracts,  performance  measures  that  accurately  judge  die  effectiveness  of 
electronic  commerce  resource  center  program  contractors  in  increasing  vendor 
use  of  electronic  commerce/electronic  data  interchange  technologies  to  conduct 
business  with  the  Federal  Government. 

e.  Select  a  type  of  contract  to  acquire  electronic  commerce  resource 
center  support  that  provides  contractors  with  the  greatest  incentive  to  employ 
efficient  and  cost-effective  methods  and  practices  while  minimizing  Government 
risk. 


f.  Revise  the  contracting  approach  for  this  program  to: 

(1)  Include  a  use-of-fee  clause  in  each  electronic  commerce 
resource  center  contract  to  require  contractors  to  justify  their  need  for  fees  in 
accordance  with  criteria  contained  in  Defense  Federal  Acquisition  Regulation 
Supplement  215.972,  "Modified  Weighted  Guidelines  Method  for  Nonprofit 
Organizations,"  and  Office  of  Management  and  Budget  Circular  A-122,  "Cost 
Principles  for  Nonprofit  Organizations. "  Fees  should  be  justified  on  all  contract 
actions  requiring  cost  analysis  and  no  less  than  annually.  The  annual  fee  request 
should  include: 
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(a)  A  description  of  each  fee  expense. 

(b)  A  statement  why  the  fee  is  not  chargeable  under 
existing  cost  guidelines. 


(c)  An  explanation  of  how  each  incurred  fee  expense 
benefits  the  electronic  commerce  resource  center  program. 


awards. 


(d)  An  annual  report  on  the  actual  use  of  prior  year  fee 


(2)  Determine  whether  prior  year  fees  were  used  in  accordance 
with  approved  fee  requests,  and  reduce  authorized  fees  for  unexpended 
balances. 


(3)  Perform  an  annual  operating  cycle  analysis  to  determine 
electronic  commerce  resource  center  fee  needs. 

(4)  Assess  alternatives  to  the  award  of  fees,  such  as  advanced 
funding  arrangements,  contractual  guarantees  for  contingencies,  providing 
Government  facilities  or  equipment,  or  others;  and  use  alternatives  when  more 
economical. 

g.  Analyze  electronic  commerce  resource  center  and  Procurement 
Technical  Assistance  Cooperative  Agreement  program  functions  and  coordinate 
electronic  commerce/electronic  data  interchange  efforts  to  eliminate  duplication. 

h.  Establish  a  Government- wide  electronic  commerce/electronic  data 
interchange  Integrated  Process  Team  to  coordinate  electronic  commerce 
resource  center  activities  with  other  Government  programs  having  similar 
missions.  As  a  minimum,  the  Integrated  Process  Team  should  attempt  to 
include  representatives  from  the  Office  of  Federal  Procurement  Policy,  the 
Department  of  Commerce,  the  Small  Business  Administration,  the  General 
Services  Administration,  the  Director,  DoD  Electronic  Commerce,  the  Defense 
Finance  and  Accounting  Service,  and  the  Military  Departments. 

i.  Streamline  the  existing  multi-layered  electronic  commerce  resource 
center  program  management  structure  by  eliminating  the  Institute  for  Defense 
Analyses,  Production  Technology,  Incorporated,  and  the  Fleet  and  Industrial 
Supply  Center,  and  use  in-house  resources  to  accomplish  their  functions. 

j.  Reduce  program  infrastructure  costs  to  a  more  reasonable  level. 

Management  Comments  and  Audit  Response.  The  Deputy  Under  Secretary 
of  Defense  (Logistics)  did  not  respond  to  a  draft  of  this  report.  Therefore,  we 
request  the  Deputy  Under  Secretary  of  Defense  (Logistics)  to  provide  comments 
by  March  12,  1997. 
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Part  n  -  Additional  Information 


Appendix  A.  Scope  and  Methodology 

Scope 


Electronic  Commerce  Resource  Program.  We  reviewed  records  and 
supporting  documentation  for  the  ECRC  program,  valued  at  $108.6  million,  for 
the  period  FYs  1993  through  1996.  Specifically,  we  evaluated  strategic  plans, 
management  plans,  program  status  reports,  contract  documents,  congressional 
language,  and  budget  and  cost  estimates  relating  to  the  ECRC  program.  We 
also  contacted  a  statistical  sample  of  ECRC  customers  to  determine  whether 
ECRCs  had  provided  effective  customer  support  and  whether  ECRCs  were 
reaching  their  target  audience.  We  defined  effective  ECRC  support  as  any  core 
function  support  tiiat  helped  a  vendor  or  Government  organization  implement  or 
increase  its  use  of  EC/EDI  technologies  to  conduct  business  with  the 
Government.  ECRCs  defined  their  target  audience  as  vendors  and  Government 
organizations  that  want  to  conduct  business  with  the  Government. 

Reliance  on  Computer-Processed  Data.  We  performed  limited  tests  on  the 
reliability  of  the  computer-processed  data  provided  by  the  ECRC  program 
management  office.  We  screened  the  ECRC  data  base  to  ensure  that  it  included 
verifiable  records  of  services  to  valid  customer  organizations.  We  also  screened 
the  ECRC  data  base  to  ensure  that  it  did  not  include  overstated  records.  We 
contacted  customers  to  confirm  that  they  had  received  the  services  recorded  in 
the  ECRC  data  base.  To  achieve  the  audit  objectives,  we  relied  on  computer- 
processed  data  in  the  ECRC  data  base.  Results  of  the  data  test  showed  an  error- 
to-problem  rate  that  raises  questions  about  the  validity  of  using  the  ECRC 
database  information  as  a  measure  of  program  performance.  However,  when 
the  data  are  reviewed  in  context  with  other  available  evidence,  we  believe  the 
opinions,  conclusions,  and  recommendations  in  this  report  are  valid. 

Audit  Period  and  Standards,  and  Locations.  We  performed  this  program 
audit  from  September  1995  through  September  1996  in  accordance  with 
auditing  standards  issued  by  the  Comptroller  General  of  the  United  States,  as 
implemented  by  the  Inspector  General,  DoD. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  within  DoD,  CAMP,  CTC,  Department  of  Commerce  (National 
Institute  of  Standards  and  Technologies),  General  Accounting  Office,  Institute 
for  Defense  Analyses,  Production  Technologies,  Incorporated,  and  Small 
Business  Administration.  Further  details  are  available  on  request. 


Statistical  Sampling  Methodology 


The  Quantitative  Methods  Division,  Inspector  General,  DoD,  developed  the 
statistical  sampling  plan  for  this  audit.  Division  personnel  statistically  selected 
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the  number  of  vendors  that  received  assistance  from  the  ECRCs.  The  statistical 
sampling  methodology  described  below  was  used  during  the  audit. 

Audit  Universe.  The  ECRC  program  management  office  gave  the  audit  team  a 
data  base  containing  51,594  records  of  ECRC  core  function  events  from 
FY  1994  through  the  first  quarter  of  FY  1996.  After  screening  the  data  base, 
we  excluded  11,177  of  the  51,594  core  function  events  reported  because  they 
were  incomplete  (see  Table  A-1  and  Appendix  D).  The  remaining 
40,417  records  in  the  data  base  represent  12,764  ECRC  customer  organizations 
that  received  EC/EDI  support  from  the  ECRCs,  which  comprise  the  audit 
universe  (see  Table  A-2). 


Table  A-1.  Adjusted  ECRC  Database 

Reported 

Percent  of 

Regional 

Database 

Incomplete 

ECRC 

Records 

Incomplete 

Verifiable 

Records 

Oakland 

4,919 

3,658 

HQH 

74 

Palestine 

5,476 

3,097 

57 

Scranton 

6,794 

2,967 

44 

San  Antonio 

3,253 

730 

2,523 

22 

Atlanta 

999 

121 

878 

12 

Orange 

392 

21 

371 

5  ■ 

Dayton 

5,129 

265 

4,864 

5 

Fairfax 

4,699 

216 

4,483 

5 

Bremerton 

3,356 

36 

3,320 

1 

Johnstown 

4,577 

20 

4,557 

1 

Cleveland 

11,976 

23 

11,953 

1 

Unknown^ 

24 

23 

1 

— 

Total 

51,594 

11,177 

40,417 

22 

Sampling  Plan.  We  used  a  stratified,  two-stage  sample  design  to  select  ECRC 
customers  from  the  audit  universe.  In  the  first  stage,  we  categorized  ECRC 
customers  by  the  number  of  different  types  of  events  received: 
ECRC  customers  receiving  only  one  type  of  core  fimction  were  included  in  the 
first  stratum;  ECRC  customers  receiving  two  types  of  core  ftmctions  were 
included  in  die  second  stratum;  and  so  on.  In  the  second  stage,  we  selected  a 
simple  random  sample  of  25  ECRC  customers  from  each  of  the  4  strata,  for  a 
total  of  100  ECRC  customers.  However,  because  of  the  large  numlier  of 
nonresponses,  we  contacted  151  organizations  to  obtain  the  100  responses  from 
ECRC  customers  discussed  in  the  report. 

Sample  Selections.  Table  A-2  lists  the  number  of  organizations  in  each 
stratum,  the  average  number  of  core  function  events  received,  the  size  of  each 


^Twenty-four  of  the  51,594  records  did  not  identify  the  regional  ECRC, 
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random  sample,  the  number  of  selections  needed  to  obtain  each  sample  size,  and 
the  number  of  nonresponses. 


Table  A-2.  Sample  Selections 

Number  of 

Average 

Size  of 

Number 

Non- 

Stratum 

Organizations 

Events 

Sample 

Needed 

Responses 

1 

10,665 

■n 

25 

43 

18 

2 

1,633 

25 

37 

12 

3 

387 

25 

38 

13 

4 

79 

mm 

25 

33 

8 

4 

12,764 

2.8 

100 

151 

51 

Survey  Questions.  We  used  a  telephone  questionnaire  to  collect 
information  from  100  ECRC  customers.  To  determine  whether  the  ECRC 
support  was  effective,  we  asked  the  customers  surveyed  whether  they 
implemented  or  increased  their  use  of  EC/EDI  technologies  as  a  result  of  the 
assistance  received  from  the  ECRCs.  To  determine  whether  ECRCs  were 
supporting  their  target  audience,  we  asked  the  customers  surveyed  whether  they 
conduct  business,  directly  or  indirectly,  with  the  Government.  To  determine 
why  some  ECRC  customers  were  not  using  EC/EDI,  we  asked  those  who  had 
not  implemented  EC/EDI,  why  they  had  not  and  if  they  planned  to  do  so. 

Sample  Results.  Table  A-3  shows  the  results  of  the  customer  survey  and  the 
range  of  projections,  including  the  best  estimate,  for  the  following  attributes: 
customers  who  did  not  respond  to  the  survey  (nonresponses);  customers  outside 
the  target  audience;  customers  within  the  target  audience,  but  not  using 
EC/EDI;  customers  within  the  target  audience  who  use  EC/EDI  and  did  not 
receive  effective  ECRC  support;  and  customers  within  the  target  audience  who 
use  EC/EDI  and  received  effective  ECRC  support. 
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Table  A-3.  Results  of  IG,  DoD,  Survey 

Attribute 

Lower 

Bound 

Best 

Estimate 

Upper 

Bound 

Best  Estimate  As 
Percent  of  Total 

Nonresponses 

51 

3,556 

5,146 

6,735 

40.3 

Outside  Target 

32 

1,069 

2,320 

3,659 

18.2 

Target, Not  Using  EC/EDI 

26 

1,439 

2,755 

4,072 

21.6 

Target,  Using  EC/EDI: 

Not  Effective 

26 

1,110 

1,441 

1,772 

mM 

Effective 

16 

173 

1,102 

2,030 

Total 

151 

12,764 

100.0 

Nonresponses.  A  significant  number  of  ECRC  customers  could  not  or 
would  not  participate  in  the  survey.  We  are  95-percent  confident  that  no  less 
than  3,556  and  no  more  than  6,735  of  the  12,764  ECRC  customers,  with 
5,146  (40  percent)  being  the  best  estimate,  would  not  participate  in  the  survey. 
TTie  ECRC  customers  surveyed  did  not  participate  because:  no  one  answered 
the  telephone,  the  telephone  line  was  disconnected,  the  customer  was  no 
longer  in  business,  the  customer  did  not  return  telephone  calls,  or  the  customer 
refused  to  participate.  Given  the  nature  of  the  nonresponses,  it  is  unlikely  the 
nonresponses  include  a  substantial  number  of  successful  EC/EDI  users. 

Outside  the  Target  Audience.  We  are  95-percent  confident  that  no  less 
than  1,069  and  no  more  than  3,659  ECRC  customers,  with  2,320  (18  percent) 
being  the  best  estimate,  do  not  conduct  business  with  the  Government. 
Consequently,  DoD  is  incurring  costs  to  assist  a  significant  number  of 
organizations  from  which  there  is  no  immediate  benefit. 

Within  the  Target  Audience  But  Not  Using  EC/EDI.  We  are 
95-percent  confident  that  no  less  than  1,439  and  no  more  than  4,072  of 
ECRC  customers,  with  2,755  (22  percent)  being  the  best  estimate,  conduct 
business  with  the  Government  but  do  not  use  EC/EDI.  Twenty-two  percent  of 
ECRC  customers  were  within  the  ECRC-defined  target  audience,  but  did  not 
implement  EC/EDI  as  a  result  of  ECRC  support.  However,  about  one-third  of 
the  22  percent  indicated  that  they  were  either  implementing  EC/EDI  or  planning 
to  implement  EC/EDI. 

Within  the  Target  Audience,  Did  Not  Attribute  Using  EC/EDI  to 
ECRCs.  We  are  95 -percent  confident  that  no  less  than  1,110  and  no  more  than 
1,772  of  the  ECRC  customers  surveyed,  with  1,441  (11  percent)  being  the  best 
estimate,  use  EC/EDI  to  conduct  business  with  the  Government.  However,  the 
11  percent  implemented  or  increased  their  use  of  EC/EDI  prior  to  receiving 
support  from  the  ECRCs  and  indicated  that  the  ECRCs  were  not  instrumental  in 
their  decision. 
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Within  the  Target  Audience,  Attribute  Using  EC/EDI  to  ECRCs. 
We  are  95-percent  confident  that  no  less  than  173  and  no  more  than  2,030  of 
the  ECRC  customers  surveyed,  with  1,102  (8.6  percent)  being  the  best  estimate, 
were  within  the  target  audience  and  implemented  or  increased  their  use  of 
EC/EDI  because  of  support  received  from  the  ECRCs.  As  a  result,  only 
8.6  percent  of  ECRC  customers  received  effective  support  from  the  ECRCs. 


30 


Appendix  B.  Summary  of  Prior  Audits  and 
Other  Reviews 


The  Office  of  the  Inspector  General,  DoD,  reports  are  shown  below.  Three 
reports  covering  issues  related  to  the  audit  of  the  Electronic  Commerce 
Resource  Centers  Program  have  been  issued. 


Inspector  General,  DoD 


Report  No.  97-007,  "Department  of  Defense  Procurement  Technical  Assistance 
Cooperative  Agreement  Program,"  October  18,  1996,  concludes  that  the 
mission  and  service  locations  of  Procurement  Technical  Assistance  Centers 
significantly  overlap  those  of  the  Small  Business  Development  Centers. 
Further,  both  organizations  provide  EC/EDI  outreach  and  education  to  vendors. 
Due  to  the  overlap,  DoD  spent  $5.6  million  in  addition  to  the  $12  million 
authorized  by  Congress  in  FY  1995  for  procurement  assistance  that  could  be 
provided  more  appropriately  and  efficiently  by  the  Small  Business 
Administration.  The  report  recommends  that  the  Director,  Defense  Logistics 
Agency,  resubmit  the  initiative  to  repeal  United  States  Code,  Title  10,  chapter 
142,  for  the  FY  1998  Defense  Authorization  Bill  to  be  proposed  by  DoD.  The 
report  also  recommends  that  the  Director,  Defense  Logistics  Agency,  give  first 
preference  for  available  Procurement  Technical  Assistance  Program  (PTAP) 
funding  to  acceptable  statewide  applicants.  The  Director,  Defense  Logistics 
Agency,  responded  stating  it  requested  repeal  of  the  program  for  FY  1997  and 
the  repeal  was  not  approved.  Resubmission  was  not  planned  because  a  repeal 
would  have  little  acceptance  in  Congress  due  to  the  Senate  and  House 
Conunittees  endorsing  the  PTAP.  The  Defense  Logistics  Agency  also  stated  it 
improved  the  PTAP  giving  greater  preference  to  statewide  programs  by  revising 
applicant  evaluation  criteria. 

Report  No.  96-064,  "Implementation  of  the  DoD  Management  Control  Program 
Within  Special  Access  Programs,"  February  2,  1996,  concludes  that  DARPA 
managers  lacked  the  necessary  management  control  training  and  did  not  conduct 
comprehensive  management  control  reviews  of  DARPA  programs.  No 
recommendations  were  made  because  Office  of  Management  and  Budget 
Circular  A-123,  "Internal  Control  Systems,"  August  4,  1986,  and  DoD 
Directive  5010.38,  "Internal  Management  Control  Program,"  April  14,  1987, 
were  revised  to  allow  management  flexibility  in  evaluating  management 
controls. 

Report  No.  95-048,  "Contracting  Practices  for  the  Use  and  Operations  of 
DoD-Sponsored  Federally  Funded  Research  and  Development  Centers," 
December  2,  1994,  assessed  contractor  justification  and  DoD  analyses  of  fee 
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requirements.  The  report  concludes  that  contracting  officers  did  not  limit  fees 
to  ordinary  and  necessary  expenses,  and  recommends  including  a  clause  in  each 
contract  to  require  contractors  to  describe  and  justify  fee  requirements.  The 
Director,  Defense  Research  and  Engineering,  agreed  and  stated  that  review  of 
fees  was  required  in  its  management  plan. 


Office  of  the  Secretary  of  Defense 


Director,  Defense  Research  and  Engineering,  study,  "Comprehensive  Review  of 
the  Department  of  Defense’s  Fee-Granting  Process  for  Federally  Funded 
Research  and  Development  Centers,"  May  1,  1995,  addresses  the  use  of  fees  by 
Federally  Funded  Research  and  Development  Centers,  the  apparent 
contradiction  between  the  need  for  fees  at  some  Federally  Funded  Research  and 
Development  Centers  and  the  lack  of  a  fee  requirement  at  others,  and 
recommendations  for  changing  the  fee  structure  for  Federally  Funded  Research 
and  Development  Centers.  The  study  supports  limiting  the  use  of  fees  paid  to 
Federally  Funded  Research  and  Development  Centers  to  reimbursement  of  costs 
that  are  ordinary  and  necessary  for  business  operations  and  assessing  the  need 
for  fees. 
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The  chart  presents  a  breakout  of  the  $76.7  million  in  funding  for  FYs  1994  and 
1995  as  allocated  across  the  ECRC  program  infrastructure.  A  brief  discussion 
of  the  responsibilities  of  each  component  of  the  infrastructure  follows. 


(FUNDS  IN  MILLIONS) 


Mndudes  $a6M  paid  to  Air  Force 
for  Program  Management. 
^Amount  Uridetermlnable. 


Figure  C-1.  Funds  Allocated  Across  ECRC  Program  Infrastructure 

Defense  Advanced  Research  Projects  Agency.  Congress  designated  DARPA 
as  the  ECRC  Program  Manager  as  of  FY  1994  with  responsibility  for  overall 
budget  authority  and  program  control.  However,  according  to  DARPA 
officials,  the  ECRC  program  did  not  fit  within  DARPA 's  mission  and  was 
considered  a  priority  only  because  of  congressional  interest.  As  a  result, 
DARPA  del^ated  ECRC  management  and  oversight  functions  to  the  other 
organizations  ,  and  had  little  day-to-day  involvement  with  the  program.  Of  the 

^The  ECRC  program  office  included  DARPA,  the  Office  of  Naval  Research, 
the  Naval  Industrial  Resources  Support  Activity,  and  the  Fleet  and  Industrial 
Supply  Center. 


33 


Appendix  C.  Funds  Allocated  Across  ECRC  Program  Infrastructure 


$76.7  million,  DARPA  transferred  $600,000  to  Air  Force  to  reimburse  them 
for  first  quarter  FY  1994  program  management  expenses,  and  transferred 
$2  million  to  the  Army  and  Rome  Laboratories  to  fund  technology  projects. 
DARPA  transferred  the  remaining  $74.1  million  among  the  following 
organizations. 

Institute  for  Defense  Analyses.  DARPA  paid  the  Institute  for  Defense 
Analyses  $600,000  to  strategically  plan  and  review  the  ECRC  program. 

Office  of  Naval  Research.  Consistent  with  the  DARPA  practice  of 
delegating  contract  support  functions,  DARPA  entered  into  an  agreement  with 
the  Office  of  Naval  Research  to  act  as  Executive  Agent  for  the  ECRC  Program. 
As  the  Executive  Agent,  the  Office  of  Naval  Research  administered  and 
distributed  ECRC  program  funds  and  delegated  program  responsibilities  to 
support  activities.  The  Office  of  Naval  Research  received  no  compensation  for 
the  functions  performed. 

Fleet  and  Industrial  Supply  Center.  The  Office  of  Naval 
Research  delegated  contracting  office  responsibility  to  the  Fleet  and  Industrial 
Supply  Center  to  award  cost-plus-fixed-fee  ECRC  contracts  and  to  perform 
post-award  contract  administration  functions.  Delegating  contracting 
responsibilities  is  a  normal  practice  for  the  Office  of  Naval  Research.  The  Fleet 
and  Industrial  Supply  Center  received  $400,000  to  conduct  these  functions. 

Naval  Industrial  Resources  Support  Activity.  The  Fleet  and 
Industrial  Supply  Center  delegated  Contracting  Officers'  Technical 
Representative  responsibilities  such  as  providing  technical  direction  and 
guidance  to  the  contracting  officer,  and  monitoring  services  performed  under 
the  ECRC  contracts,  to  the  Naval  Industrial  Resources  Support  Activity.  The 
Naval  Industrial  Resources  Support  Activity  received  approximately  $50,000  to 
cover  travel  expenses  for  the  Contracting  Officers'  Technical  Representative. 

Production  Technology,  Incorporated.  Because  the  Naval 
Industrial  Resources  Support  Activity  had  no  in-house  staff  for  such  purposes, 
the  Office  of  Naval  Research  paid  Production  Technology,  Incorporated, 
$750,000  to  perform  Contracting  Officers'  Technical  Representative  functions 
such  as  technical  reviews  of  contract  deliverables  and  preparing  ECRC  activity 
reports. 

Team  Integrators.  Congress  directed  DoD  to  award  sole-source 
contracts  to  CTC  and  CAMP  as  ECRC  team  integrators  to  coordinate,  plan,  and 
oversee  the  activities  of  the  regional  ECRCs  with  direction  from  the 
ECRC  program  office. 

Concurrent  Technologies  Corporation.  CTC,  a  not-for-profit 
organization,  headquartered  in  Johnstown,  Pennsylvania,  has  direct  contract 
award  authority  as  well  as  management  and  oversight  responsibility  for  six 
regional  ECRCs.  Three  of  the  six  regional  ECRCs  are  operated  by 
CTC  employees  with  subcontractor  support.  The  remaimng  three  ECRCs  are 
operated  entirely  by  subcontractors.  CTC  retained  $1  million  to  pay  for  team 
integrator  functions. 


34 


Appendix  C.  Funds  Allocated  Across  ECRC  Program  Infrastructure 


Cleveland  Advanced  Manufacturing  Program.  CAMP,  a 
not-for-profit  organization,  located  in  Cleveland,  Ohio,  coordinated  activities 
for  five  ECRCs,  however,  CAMP  did  not  have  contract  award  authority  or 
direct  management  responsibility  for  these  ECRCs.  Funding  for  the 
CAMP  ECRCs  flowed  through  the  Fleet  and  Industrial  Supply  Center,  which 
competitively  awarded  the  contracts  for  operation  of  the  regional  ECRCs.  Like 
CTC,  CAMP  obtained  direct  contract  award,  and  management  and  oversight 
authority  for  its  regional  ECRCs  when  the  program  shifted  to  the  Deputy  Under 
Secretary  of  Defense  (Logistics)  in  FY  1997.  CAMP  retained  $1.3  million  to 
pay  for  team  integrator  functions. 

Regional  ECRCs  and  Subcontractors.  The  two  regional  ECRCs  use 
subcontractors  to  accomplish  their  mission.  Each  ECRC  is  required  to  form  a 
partnership  with  a  not-for-profit  organization  (such  as  a  university)  to 
accomplish  its  mission.  For  example,  die  Fairfax  ECRC,  which  is  operated  by 
Dimensions  International,  Incorporated,  formed  partnerships  with  two 
subcontractors:  Iris,  Limited  Liability  Partnership;  and  George  Mason 

University.  Iris  and  George  Mason  perform  ECRC  ftinctions  for  Dimensions 
including  providing  outreach,  education,  and  training.  In  addition,  the  Fairfax 
ECRC  collaborated  with  the  local  Small  Business  Development  Center  and 
Manufacmring  Extension  Providers  to  accomplish  outreach.  As  another 
example,  the  Atlanta  ECRC  is  jointly  operated  by  Georgia  Tech  University  and 
Clark  Atlanta  University.  Of  the  $49.2  million  paid  to  die  regional  ECRCs,  we 
were  unable  to  separately  identify  funds  retained  by  the  11  ECRCs  and  fUnds 
paid  to  subcontractors. 
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ECRC  Use  of  Activity  Metrics.  The  ECRCs  use  activity  metrics  to  measure 
the  effectiveness  of  the  ECRC  program.  Activity  metrics  are  the  number  of 
core  Unction  events  (instances  of  outreach,  education  and  training,  consultation, 
and  technical  support)  die  11  regional  ECRCs  accomplished  during  a  given 
period.  In  December  1995,  we  requested  a  consolidated  data  base  of  regional 
ECRC  activity  metrics  from  the  beginning  of  FY  1994  through  the  first  quarter 
of  FY  1996.  The  team  integrators  and  the  ECRC  program  office  took  almost 
3  months  to  provide  the  requested  information.  In  March  1996,  the  ECRC 
program  office  provided  11  data  bases  of  core  function  events  conducted  by  the 
11  regional  ECRCs.  We  consolidated  the  data  bases  and  counted  51,594  ECRC 
core  function  events  during  the  27  month-period,  over  1,900  events  per  month. 

ReliabUity  and  Effectiveness  of  ECRC  Activity  Metrics.  To  determine  the 
reliability  of  ECRC  activity  metrics,  the  Inspector  General,  DoD,  Audit, 
screened  the  51,594  records  to  ensure  that  each  record  included  sufficient 
information  to  verify  the  reported  event  with  the  customer  organization.  To 
determine  the  effectiveness  of  ECRC  core  functions,  we  questioned  a  statistical 
sample  of  ECRC  customers  concerning  the  ECRC  support  they  received. 
Effective  ECRC  support  consisted  of  one  or  more  core  function  events  that 
helped  a  customer  organization  implement  or  increase  its  use  of  EC/EDI 
tec^ologies  to  conduct  business  with  the  Federal  Government. 

Incomplete  Records.  We  determined  that  11,177  of  the 
51,594  records,  or  21  percent,  were  not  complete  because  they  did  not  identify 
the  organization,  describe  the  event,  or  indicate  the  core  function  provided. 
Three  ECRCs  (OaklaiKi,  Palestine,  and  Scranton)  accounted  for  87  percent  of 
the  incomplete  records.  The  remaining  40,417  records  were  verifiable  in  that 
they  contained  sufficient  information  to  test  the  accuracy  of  reported  activity 
metrics. 


Overstated  Events.  We  determined  that  9,464  of  the 
40,417  verifiable  records,  or  23  percent,  were  overstated.  We  maintained  that 
ECRC  customer  organizations  received  only  one  core  function  event  per  day, 
and  therefore,  multiple  core  functions  per  day  were  considered  to  be 
overstatements.  We  tested  die  40,417  verifiable  records  for  duplication  and 
found  that  identical  records  were  entered  more  than  one  time,  and  instances 
where  the  ECRCs  claimed  more  than  one  core  function  event  for  the  same 
recipient  on  the  same  date.  For  example,  an  ECRC  reported  that  36  East  Ohio 
Gas  Company  employees  attended  4  events  each  on  August  9,  1994.  The 
ECRC  report  144  records  for  this  incident  (36  employees  x  4  events). 
According  to  East  Ohio  Gas  Company,  the  ECRC  conduct^  one  seminar  on 
that  date  for  20  company  employees.  Although  the  ECRC  received  credit  for 
144  events  (records),  it  should  have  only  recorded  1  event.  Two  ECRCs 
(Fairfax  and  Clevel^)  accounted  for  74  percent  of  the  overstated  records. 
Outreach,  consultation,  and  education  events  comprised  97  percent  of  the 
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overstated  records.  Overall,  40  percent  of  the  ECRC  data  base  records  were 
incomplete  or  overstated,  therefore,  only  60  percent  of  the  records  were  reliable 
for  audit  purposes. 

Table  D-1  shows  the  number  of  reported,  incomplete,  verifiable,  overstated, 
and  remaining  ECRC  data  base  records. 


Table  D-1.  Reliability  of  ECRC  Activity  Metrics 

Regional 

ECRC 

Reported 

Database 

Records 

Incomplete 

Verifiable 

Over¬ 

stated 

Remaining 

Records 

Percent  of 
Remaining 
Records 

Oakland 

4,919 

3,658 

1,261 

99 

1,162 

24 

Palestine 

5,476 

3,097 

2,379 

299 

2,080 

38 

Fairfax 

4,699 

216 

4,483 

2,611 

1,872 

40 

Scranton 

6,794 

2,967 

3,827 

127 

3,700 

54 

Cleveland 

11,976 

23 

11,953 

4,425 

7,528 

63 

Bremerton 

3,356 

36 

3,320 

883 

2,437 

73 

San  Antonio 

3,253 

730 

2,523 

87 

2,436 

75 

Atlanta 

999 

121 

878 

4 

874 

87 

Dayton 

5,129 

265 

4,864 

403 

4,461 

87 

Johnstown 

4,577 

20 

4,557 

522 

4,035 

88 

Orange 

392 

21 

371 

4 

367 

94 

Unknown 

24 

23 

1 

0 

1 

— 

Total 

51,594 

11,177 

40,417 

9,464 

30,953 

60 

Evaluation  of  ECRC  Effectiveness.  We  contacted  a  statistical  sample 
of  151  ECRC  customer  organizations  to  determine  the  effectiveness  of  ECRC 
core  functions.  Specifically,  we  asked  each  customer  whether  an  ECRC  had 
helped  them  implement  or  increase  their  use  of  EC/EDI  technologies  to  conduct 
business  with  the  Government.  A  third  of  the  ECRC  customers  in  the  survey 
could  not  be  reached  by  telephone  or  refused  to  participate.  We  determined, 
with  95-percent  confidence,  that  about  1,102  of  the  12,764  ECRC  customers,  or 
8.6  percent,  received  effective  ECRC  support.  See  Appendix  A  for  a  discussion 
of  sample  selection  methods  and  a  detailed  discussion  of  survey  results. 

Evaluation  of  ECRC  Targeting.  According  to  the  ECRC  Operational 
Plan,  dated  March  22,  1995,  the  ECRC  target  audience  consisted  of  small-to- 
medium-sized  enterprises  that  conduct  business  with  the  Government.  To 
determine  whether  ECRCs  were  supporting  their  target  audience,  we  asked  each 
ECRC  customer  if  they  conducted  business  with  the  Government,  with  or 
without  EC/EDI  technologies.  We  determined,  with  95-percent  confidence, 
that  2,320  of  12,764  ECRC  customer  organizations,  about  18  percent,  were 
outside  the  ECRC-defined  target  audience.  Of  the  100  ECRC  customers 
responding  to  the  survey,  68  were  within  the  ECRC-defined  target  audience  and 
32  were  not.  Although  5  of  the  32  respondents  had  implemented  or  increased 
their  use  of  EC/EDI  as  a  result  of  ECRC  support,  the  5  did  not  conduct  business 
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with  the  Government;  therefore,  we  did  not  include  them  with  the  number 
receiving  effective  ECRC  support.  Figure  D-1  shows  the  distribution  of  the 
100  survey  respondents  within  and  outside  the  target  audience. 


Figure  D-1.  ECRC  Targeting 

Unclear  DeHnition  of  Target  Audience.  The  DARPA  ECRC 
program  manager  provided  guidance  to  regional  ECRCs  to  define  their  target 
audience.  Specifically,  the  program  manager  stated  that  the  target  audience  is 
the  "future  integrated  conunercial-defense  industrial  base  that  DoD  acquisition 
reform  and  dual  use  initiatives  are  opening  as  a  future  source  of  supply."  This 
definition  is  unclear  and  very  broad,  thus  not  ensuring  that  ECRC  resources 
were  focused  on  the  appropriate  target  audience.  The  Deputy  Under  Secretary 
of  Defense  (Logistics)  and  DLA  officials  need  to  clearly  identify  a  target 
audience  that  will  result  in  the  most  efficient,  cost-effective  use  of  ECRC 
resources. 

Confirmation  of  Testing  Results.  In  January  1996,  as  a  subtask  of  the 
instructional  technology  development  initiative,  the  ECRC  technology  hub 
completed  a  study  of  the  adequacy  of  ECRC  training  courseware.  The  study 
included  an  analysis  of  the  impact  of  training  provided  by  the  regional  ECRCs 
on  the  business  practices  of  training  recipients.  The  study  concluded  that, 
although  ECRC  training  courses  helped  vendors  learn  about  EC/EDI,  the 
training  did  not  result  in  significant  vendor  implementation  of  EC/EDI 
technologies.  Of  the  vendors  surveyed  by  the  hub,  83  percent  thought  the 
training  was  useful,  but  only  10  percent  of  them  used  EC/EDI  to  conduct 
business  with  the  Government  after  attending  die  training.  The  study  also 
concluded  that  ECRC  data  bases  contained  inaccurate  and  duplicate  data.  Some 
ECRC  class  counts  reported  in  the  data  bases  "included  everyone  who  intended 
to  attend  the  class,  and  not  actual  attendees. " 
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ECRC  Use  of  Impact  Metrics  to  Measure  Program  Effectiveness.  The 
ECRCs  also  use  impact  metrics  to  measure  the  effectiveness  of  the  ECRC 
program.  Impact  metrics  are  customer  success  stories  and  business  case 
analyses  that  chronicle  the  economic  benefits  gained  from  ECRC  customers 
implementing  EC/EDI  technologies  because  of  ECRC  support.  Eight  of 
11  ECRCs  reported  239  success  stories  between  FY  1994  and  the  first  quarter 
of  FY  1996,  about  9  success  stories  a  month. 

Reliability  and  Validity  of  ECRC  Impact  Metrics.  To  determine  the 
reliability  of  ECRC  impact  metrics,  we  tested  all  239  success  stories  for 
completeness  and  compliance.  Tests  of  completeness  ensured  that  each  success 
story  included  adequate  information  about  each  ECRC  customer.  Tests  of 
compliance  ensured  that  each  success  story  complied  with  the  definition  of 
success,  according  to  the  ECRC  statement  of  work,  dated  September  29,  1995, 
and  involved  an  organization  within  the  ECRC  target  audience. 

Tests  of  Reliability.  We  reviewed  all  239  documented  success  stories 
provided  by  the  8  of  the  1 1  regional  ECRCs  for  completeness  and  compliance. 

Tests  of  Completeness.  We  reviewed  ail  239  success  stories 
provided  by  the  regional  ECRCs  to  determine  whether  we  could  identify  the 
ECRC  customer,  Ae  support  received,  economic  benefits  gained,  need  for 
EC/EDI  technologies,  and  the  role  of  the  responsible  ECRC.  Half  of  the 
239  success  stories  were  not  documented  and  thus  were  incomplete.  Three 
ECRCs  (Fairfax,  Dayton,  and  San  Antonio)  accounted  for  all  but  one  of  the 
incomplete  success  stories. 

Tests  of  Compliance.  We  reviewed  all  119  complete  success 
stories  to  determine  whether  the  ECRC  customers  implemented  EC/EDI 
technologies  with  the  help  of  an  ECRC  and  conducted  business  with  the 
Government.  We  did  not  attempt  to  quantity  the  economic  benefits  gained  by 
the  ECRC  customer  due  to  lack  of  documentation.  We  determined  that  31  of 
the  119  success  stories,  or  26  percent,  involved  valid  implementations  of 
EC/EDI  technologies  for  the  purpose  of  conducting  business  with  the  Federal 
Government.  The  remaining  88  success  stories  were  not  valid  because  the 
customers  either  did  not  implement  EC/EDI  technologies,  or  did  not  conduct 
business  with  the  Federal  Government.  Overall,  31  of  239  ECRC  success 
stories,  or  13  percent,  were  not  reliable. 
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Table  E-1  shows,  by  region,  the  results  of  our  analysis  of  ECRC  impact  metrics 
(success  stories). 


Table  E-1.  Analysis  of  ECRC  Impact  Metrics 

Regional 

ECRC 

Reported 

Impact 

Metrics 

Incomplete 

Complete 

Not 

Compliant 

Reliable 

Reliable 
as  Percent 
Reported 

Dayton 

33 

33 

0 

0 

0 

0 

Oakland 

0 

0 

0 

0 

0 

0 

Orange 

0 

0 

0 

0 

0 

0 

San  Antonio 

28 

28 

0 

0 

0 

0 

Scranton 

0 

0 

0 

0 

0 

0 

Johnstown 

26 

0 

26 

24 

2 

8 

Palestine 

37 

1 

36 

31 

5 

14 

Cleveland 

34 

0 

34 

23 

11 

32 

Atlanta 

6 

0 

6 

4 

2 

33 

Fairfax 

62 

58 

4 

2 

2 

50 

Bremerton 

13 

0 

13 

4 

9 

69 

Total 

239 

120 

119 

88 

31 

13 

Tests  of  Validity.  We  judgmentally  contacted  10  of  the  119  customers 
from  the  complete  success  stories  to  determine  whether  their  success  stories 
were  valid.  We  reviewed  ECRC  support  documentation  and  interviewed 
company  officials.  Success  stories  were  valid  success  stories  if  the  customer 
implemented  EC/EDI  technologies  to  conduct  business  with  the  Government 
and  received  productive  support  from  an  ECRC.  We  determined  that  only  3  of 
the  10  customers  were  valid  success  stories.  Therefore,  the  success  stories  did 
not  accurately  reflect  the  effectiveness  of  ECRC  core  functions. 

Valid  Success  Stories.  We  considered  the  following  success 
stories  to  be  valid  because  the  customers  implemented  EC/EDI  technologies  to 
do  business  with  the  Government. 

Laurel  Technologies,  Incorporated.  Laurel 

Technologies  designs  and  manufactures  shipboard  display  systems  for 
commercial  and  Government  use.  From  March  1994  through  February  1995, 
the  Johnstown  ECRC  provided  more  than  1,000  free  hours  of  support  to  help 
Laurel  Technologies  develop  and  implement  an  on-line  communication  system. 
As  a  result  of  the  Johnstown  ECRC  support,  laurel  Technologies  is  able  to 
electronically  transfer  engineering  files  to  a  DoD  prime  contractor,  improve 
customer  service,  and  reduce  paperwork. 

D-Bolt  Company,  Incorporated.  The  D-Bolt  Company 
distributes  metal  fasteners  to  the  steel  industry  and  to  Government 
subcontractors.  The  Johnstown  ECRC  helped  the  D-Bolt  Company  implement 
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EDI  and  improve  customer  service.  As  a  result  of  the  Johnstown  ECRC 
support,  D-Bolt  Company  won  several  commercial  contracts  and  is  better 
positioned  to  electronically  pursue  more  Government  business. 

Distal  Imaging  of  Oviedo,  Florida.  The  company 
develops  state-of-the-art  infrared  imaging  systems  for  military  and  commercial 
uses  and  markets  video  products  through  an  affiliated  company.  The  Atlanta 
ECRC  provided  Digital  Imaging  with  four  hours  of  support  to  evaluate  EDI 
software.  As  a  result  of  the  Atlanta  ECRC  support.  Digital  Imaging  purchased 
EDI  software  and  implemented  EDI  through  its  affiliated  company. 

Invalid  Success  Stories.  We  considered  the  following  success 
stories  to  be  invalid  because  the  customers  did  not  implement  EC/EDI 
technologies  to  do  business  with  the  Government. 

All  Water  Systems.  All  Water  Systems  designs,  sells, 
and  services  high-purity  water  filtration  systems  for  commercial  and 
Government  use.  All  Water  Systems  contacted  the  Johnstown  ECRC  in 
November  1995  to  become  EC/EDI  compatible  and  establish  an  electronic 
catalog.  The  Johnstown  ECRC  provided  All  Water  Systems  with  technical 
support  to  develop  a  data  management  conversion  plan  and  demonstrate 
potential  EC/EDI  solutions.  The  Johnstown  ECRC  described  All  Water 
Systems  as  a  success  because  the  company  was  "implementing  EDI/Legacy  Data 
Management  technologies  in  the  first  and  second  quarter  of  1996."  As  of  June 
1996,  All  Water  Systems  had  upgraded  its  computer  equipment  and  planned  to 
purchase  EC/EDI  software  in  the  near  future.  However,  All  Water  Systems  had 
not  implemented  EC/EDI  to  do  business  with  the  Federal  Government  as 
reported  by  the  regional  ECRC. 

Jet  Aerospace,  Incorporated.  Jet  Aerospace 
manufactures  precision  machine  components  for  a  DoD  prime  contractor.  In 
FY  1995,  the  Johnstown  ECRC  provided  Jet  Aerospace  with  20  hours  of 
EC/EDI  support  to  develop  price  quotes  and  process  purchase  orders.  The 
Johnstown  ECRC  described  Jet  Aerospace  as  a  success  because  the  company 
was  "implementing  EDI  in  the  first  quarter  of  1996."  As  of  June  1996,  Jet 
Aerospace  had  not  implemented  EDI,  but  credited  the  ECRC  for  preparing  the 
company  to  do  so  in  the  future. 

Dynamic  Design  Engineering.  Dynamic  Design 
Engineering  provides  consulting  and  engineering  services  but  does  not  conduct 
business  with  the  Government.  In  August  1995,  the  company  received  training 
from  the  Johnstown  ECRC  to  learn  about  business  opportunities  with  DoD 
through  EDI.  The  Johnstown  ECRC  subsequently  reported  the  company  was 
"implementing  EDI  (in  early)  FY  1996."  According  to  the  company's 
President,  the  company  implemented  EDI  prior  to  contacting  the  ECRC  and  did 
not  benefit  from  the  ECRC  support. 

Tranzonic  Companies.  Tranzonic  Companies 
manufactures  non-electronic  hospital  and  safety  supplies  but  does  not  conduct 
business  with  the  Federal  Government.  The  Cleveland  ECRC  provided 
Tranzonic  Company  with  24  free  hours  of  manufacturing  and  marketing 
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assistance,  but  no  EC/EDI  support.  The  Cleveland  ECRC  claimed  success  for 
helping  Tranzonic  "enter  a  new  market  and  position  itself  with  a  significant 
competitive  advantage  over  any  existing  product  on  the  market. "  According  to 
the  company's  Vice  President,  the  company  has  not  implemented  EC/EDI  or 
benefitted  from  the  ECRC's  manufacturing  and  marketing  support. 

Youngstown  Forge,  Incorporated.  Youngstown  Forge 
produces  forged  steel  and  alloy  products,  Wt  does  not  conduct  business  with  &e 
Federal  Government.  The  Cleveland  ECRC  provided  the  company  with  4  free 
hours  of  support,  then  included  the  company  on  its  list  of  success  stories 
because  it  was  actively  seeking  DoD  business.  According  to  the  company's 
President,  Youngstown  Forge  has  not  made  the  investment  to  implement  EDI 
but  is  working  toward  doing  so,  and  hopes  to  do  business  with  the  Federal 
Government. 


The  Supply  Room.  The  Supply  Room  sells  office 
supplies  to  commercial  and  Government  organizations  using  non-EC/EDI 
methods.  In  1995,  the  company  received  an  hour  and  a  half  consultation  from 
the  Fairfax  ECRC  to  discuss  EDI  software  applications  and  potential  trading 
partners.  Although  the  company  successfully  implemented  EC/EDI  to  do 
business  with  the  Federal  Government,  they  implemented  EDI  prior  to  receiving 
support  from  the  Fairfax  ECRC,  and  discontinued  using  it  in  April  1996 
because  it  was  not  cost  effective. 

Rayco  Supply  Company.  Rayco  Supply  Company 
provides  industrial  marine  products  to  DoD  and  requested  EC/EDI  assistance 
from  the  Fairfax  ECRC.  The  company  received  4  hours  of  support  from  the 
Fairfax  ECRC  to  "become  EDI  literate,"  and  in  selecting  a  Value  Added 
Network.  According  to  a  Rayco  Supply  official,  they  attributed  their 
implementation  of  EC/EDI  to  a  third  pai^  vendor,  and  not  to  the  support 
received  from  the  Fairfax  ECRC. 
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The  Hub  undertook  15  initiatives  during  FYs  1995  through  1996  to  accomplish 
its  mission  to  identify  and  deploy  new  and  existing  EC/EDI  technologies  to  help 
vendor  and  Government  organizations  implement  and  increase  their  use  of 
EC/EDI  to  do  business.  A  technical  expert  from  the  Office  of  the  Inspector 
General,  DoD,  performed  a  review  of  the  15  Hub  initiatives  to  determine  the 
extent  to  which  the  initiatives  benefited  DoD.  The  technical  reviewer  concluded 
that  the  initiatives  did  not  identify  solutions  to  vendor  and  Government 
organizations'  impediments  to  using  EC/EDI  technologies.  Four  of  the 
15  initiatives  (3  CALS  Expo  initiatives  and  the  Collaborative  Application 
Protocol  Development  Initiative)  are  discussed  in  Part  I  of  the  report.  A  brief 
discussion  of  the  remaining  11  initiatives  follows. 

Implementation  Planning  for  the  National  ECRC  Technolo^  Hub.  This 
initiative  provided  for  establishment  of  the  hub  and  developing  implementation 
plans  for  hub  initiatives.  The  budget  for  this  initiative  was  $378,654.  The 
initiative  ended  with  the  successful  planning  for  establishment  of  the  hub  in 
FY  1995. 

Electronic  Commerce  Testbed  Initiatives.  There  were  two  testbed  initiatives; 
one  in  FY  1995,  and  one  in  FY  1996.  The  FY  1995  initiative  established  the 
Hub  facility  at  CTC  to  provide  the  ECRC  program  with  capabilities  to  evaluate, 
demonstrate,  and  validate  new  technologies,  and  put  them  into  usable  products 
for  vendors.  The  FY  1996  initiative  continues  hub  efforts  to  evaluate, 
demonstrate,  and  validate  new  EC/EDI  technologies.  The  budgets  for  these 
initiatives  were  about  $4.5  million  in  FY  1995,  and  $2.7  million  for  FY  1996. 
According  to  DARPA  ECRC  program  office  officials,  the  testbed  is  of  no 
benefit  to  the  ECRC  Program  because  the  initiatives  undertaken  by  the  hub  do 
not  focus  on  vendor  EC/EDI  needs  and  are  too  technical.  The  IG,  DoD, 
technical  review  concurred  with  the  DARPA  ECRC  program  office  that  there  is 
no  need  for  the  Electronic  Commerce  Testbed  Initiatives  because  the  vendor 
technology  needs  have  not  been  identified. 

ECRC  Technology  Hub  Activities  Initiative.  In  addition  to  the  development 
of  a  World-Wide  Web  Home  Page^  and  the  EC/EDI  technical  information 
repository,  this  initiative  includes  identifying  technology  gaps  that  cannbt  be 
met  with  existing  or  emerging  technologies,  and  the  development  and  printing 
of  ECRC  brochures  and  newsletters.  The  budget  for  this  initiative  is  $505,687. 
This  initiative  duplicates  the  mission  requirement  for  regional  ECRCs  to  identify 
technology  gaps  preventing  vendors  and  Government  organizations  from  using 
EC/EDI  technologies.  In  addition,  many  software  and  hardware  developers  and 
integrators  invest  significant  resources  to  identify  and  fill  technology  voids. 
The  brochures,  newsletters,  and  related  ECRC  marketing  materials  are  of 


^The  ECRC  World  Wide  Web  site  address  is:  htq);//www.ecrc.ctc.com. 
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questionable  benefit  because  they  have  not  resulted  in  an  appreciable  increase  in 
the  use  of  EC/EDI  to  conduct  business  with  DoD,  and  should  not  be  funded 
under  the  hub. 

Instructional  Technology  Development  Initiative.  This  initiative  is  to 
evaluate  and  standardize  the  ECRC  training  curriculum,  identify  new  training 
techniques,  prioritize  the  training  provided,  and  establish  competency  levels  for 
persons  receiving  training.  The  FY  1995  budget  for  this  initiative  was 
$951,720.  ECRC  program  office  officials  consider  part  of  this  initiative  to  be 
useful;  specifically,  the  instructional  needs  assessments.  However,  we  question 
the  appropriateness  of  this  being  a  hub  initiative  since  it  duplicates  req^uirements 
in  regional  ECRC  and  team  integrator  missions.  Specifically,  team  integrators 
are  required  to  combine  the  individual  needs  of  the  ECRCs  and  use  that 
information  to  develop  a  coordinated  ECRC  training  plan.  Also,  since  regional 
ECRCs  interface  with  vendors  and  DoD  organizations,  they  are  in  the  best 
position  to  identify  gaps  in  ECRC  training. 

Knowledge  Transfer  Initiative.  This  initiative  continues  the  Instructional 
Technology  Development  Initiative  of  FY  1995.  The  objective  is  to  conduct 
instructional  needs  assessments  and  provide  courseware  configuration 
management.  The  FY  1996  budget  for  the  Knowledge  Transfer  Initiative  is 
$300,000.  The  Instructional  Technology  Development  Initiative  was  scaled 
back  in  FY  1996  eliminating  the  evaluation  and  demonstration  of  advanced 
training  tools.  TTie  Knowledge  Transfer  Initiative  was  determined  to  be 
ineffective  in  promoting  DoD  EC/EDI  goals  and  objectives.  Also,  this 
initiative  duplicates  the  requirement  in  the  statement  of  work  for  the  regional 
ECRCs  and  team  integrators  to  assess  vendor  and  Government  training  needs. 

Collaboration  with  NIST,  CALS  Industry  Steering  Group  (CALS  ISG), 
and  U.S.  Product  Data  Association.  The  objective  of  this  initiative  is  to 
maintain  close  working  relationships  with: 

o  the  National  Institute  of  Standards  and  Technologies  National 
Advanced  Manufacturing  Testbed  in  developing  engineering  technologies 
promoting  EC. 

o  the  CALS  Industry  Steering  Group  to  assure  cooperation  in  achieving 
common  goals  such  as  outreach,  education  and  training,  and  dissemination  of 
information  to  CALS  Industry  Steering  Group  members; 

0  the  U.S.  Product  Data  Association  in  the  area  of  product  data 
exchange  using  U.S.  Product  Data  Association  as  a  vehicle  to  deploy 
technology  developed  by  the  hub. 

The  budget  for  this  initiative  is  $500,890.  The  ECRCs  are  to  provide  frequent 
reports  to  assure  wide  dissemination  of  the  results  of  these  liaisons,  but  to  no 
specified  audience  and  for  no  specified  purpose.  This  initiative  appears  to  serve 
no  purpose  that  is  of  benefit  to  the  Federal  Government  because  it  does  not 
accomplish  the  hub  mission  to  develop  and  deploy  EC/EDI  technologies. 
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EDI  Integration.  The  objective  of  this  initiative  is  to  identify  software  tools, 
and  provide  guidelines  to  enable  vendors  to  integrate  American  National 
Standards  Institute  XI 2  (ANSI  XI 2)  EDI  transaction  sets  with  their  internal 
application  software.  The  budget  for  this  initiative  is  $431,041.  Specifically, 
this  initiative  was  to  identify  a  cost-effective,  easy  method  for  vendors  to  read 
data  that  is  in  ANSI  X12  format  directly  into  their  existing  internal  systems. 
According  to  an  official  from  the  Microsoft  Technical  Information  Department, 
Development  Tools  and  Languages  Group,  many  unique  translation  software 
packages  exist,  but  there  is  no  one  product  that  will  interface  all  internal 
application  software  packages  with  ANSI  X12.  Because  internal  application 
software  is  unique  to  each  vendor,  and  translation  software  packages  are  limited 
to  unique  algorithms,  each  vendor  would  have  to  build  or  adapt  software 
interfaces  compatible  with  their  respective  internal  application  software. 
Therefore,  there  is  no  single  solution  to  accommodate  all  of  the  possible 
combinations  of  internal  software  packages.  As  a  result,  the  hub  initiative  is  not 
practical.  Aso,  the  major  application  software  developers  are  building  ANSI 
X12  translation  capabilities  into  their  products.  For  example,  Microsoft  and 
Word  Perfect  software  packages  already  accommodate  importing  and  exporting 
text  in  ANSI  X12  formats.  Since  software  developers  are  embedding  ANSI 
X12  translators  in  their  products,  this  initiative  is  not  necessary. 

EC  In  Support  of  Shipyards.  The  objective  of  this  initiative,  identified  by  the 
Contracting  Officer's  Technical  Representative,  is  for  the  hub  to  enhance  and 
maintain  an  electronic  communications  network,  NSnet,  that  is  being  developed 
under  the  MARITECH  program  for  the  maritime  community.  The  NSnet  is  to 
become  the  maritime  industry's  infrastructure  for  performing  EC  and 
communications.  The  budget  for  this  initiative  is  $1,(X)0,000.  The  Hub  will 
also  develop  scenarios  and  demonstrations  of  EC  technologies,  and  provide 
outreach,  education,  and  training  to  promote  EC  in  the  shipbuilding  industry. 
According  to  the  technical  reviewer,  the  premise  here  is  that  the  principle 
shipbuilding  customer,  the  Navy,  will  develop  an  EC  requirement,  then  the 
shipbuilding  industry  will  receive  help  in  developing  the  capability  to  meet 
the  new  requirement.  However,  it  is  unlikely  that  the  Navy  has  not  recognized 
a  need  for  EDI  and  developed  a  shipboard  EC  requirement  on  its  own.  While 
the  focus  of  this  initiative  is  at  least  DoD-related,  it  is  not  necessary. 

Collaborative  Tools  for  Near-Net  Shape  Manufacturing.  Forging,  casting, 
and  powder  metallurgy  manufacturing  are  classified  as  near-net  shape  processes. 
This  initiative  integrates  EC  technologies  into  the  National  Center  for 
Excellence  in  Metalworking  Technology  initiatives,  also  operated  by  Concurrent 
Technologies,  Corporation.  The  budget  for  this  initiative  is  $750,000.  The 
objective  is  to  create  a  virtual  enterprise  environment  for  Near-net  shape 
manufacturers  for  demonstration  at  the  FY  1996  CALS  Exposition.  According 
to  the  technical  reviewer,  neither  DoD  nor  the  Government  should  be  pushing 
the  development  of  particular  STEP  Application  Protocols.  DoD  should  let  the 
marketplace  decide  on  this  issue.  Also,  this  initiative  does  not  accomplish  the 
Hub  mission  because  it  is  not  relevant  to  the  majority  of  vendors. 

Manufacturing  Product  Data  Standards  Initiative.  This  initiative  focused  on 
defining,  developing,  and  implementing  STEP  application  protocols  in  near-net 
shape  parts  manufacturing  processes.  The  budget  for  this  initiative  was  about 
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$1.6  million.  According  to  ECRC  program  officials,  this  initiative  was  of  no 
benefit  to  DoD  because  it  was  not  applicable  to  the  majority  of  vendors  and 
focused  on  manufacturing  processes  rather  than  EC/EDI.  Our  technical 
reviewer  concluded  the  same.  STEP  is  only  one  of  the  product  data  standards 
in  the  marketplace  and  is  not  the  most  heavily  used  of  its  type;  the  Initial 
Graphic  Exchange  Specification  is  more  common.  Both  standards  require  the 
development  of  application  protocols  in  order  to  be  used  in  specific  industrial 
processes.  However,  there  is  no  need  for  DoD  to  sponsor  this  effort  because 
the  marketplace  has  produced  numerous  different  interchange  standards,  and 
DoD  should  let  the  marketplace  decide.  This  initiative  was  discontinued  in 
FY  1995.  However,  the  concepts  are  being  pursued  in  FY  1996  under  the 
"Collaborative  Tools  for  Near-Net  Shape  Manufacturing"  initiative. 
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Other  Defense  Organizations 


Director,  Defense  Advanced  Research  Projects  Agency 
Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Information  Systems  Agency 
Director,  Defense  Logistics  Agency 
Inspector  General,  Defense  Intelligence  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 


Non-Defense  Federal  Organizations 


Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs,  General 
Accounting  Office 

Chairman  and  ranking  minority  member  of  each  of  the  following  congressional 
committees  and  subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subconunittee  on  Defense,  Conunittee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
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